Thread safety and Pytest

[Liam-DeVoe]

(This is a high-level issue that proposes making pytest more thread safe; I'm definitely looking for feedback, discussion, and confirmation this is a good direction here.)

Background

This issue comes out of a brief conversation on the pytest discord:

[@ngoldbaum] out of curiosity, if there were funding for work on improving pytest’s thread safety, is that something that the project might be interested in? For the work we’ve been doing so far we’ve contracted David Woods to work on Cython, David Hewitt to work on PyO3, and Liam DeVoe to work on Hypothesis. Is there an opportunity for pytest as well?

[@The-Compiler] Personally I'm not too interested in it being thread-safe, but I'd absolutely support using our Opencollective funds to make it happen

[@RonnyPfannschmidt] my sentinent is the same - but i also want to note that i'd prefer thread safety in a manner that uses queues and non-shared state for fixtures & co (aka xdist-alike, just without separate node objects) - the key contention would be log/stdio capture state + managing pytest stores on nodes in a controlled manner

I don't presume to know the best direction for thread-safety in pytest, so I'm opening this issue to get some feedback on my understanding.

While this may be obvious to all involved, I want to at least give a brief motivation for thread-safety here. Python is working towards removing the global interpreter lock, in a python build version called free-threading. As free-threading progresses, it seems likely that more people will want to run multiple tests simultaneously in separate threads. In particular, it would be nice for free-threading development if pytest-xdist supported threading (in place of multiprocessing) as its worker primitive, so existing test suites can be run with pytest-xdist in threading mode to flush out concurrency bugs.

Proposal

I would summarize all of pytest as two (very complex) steps:

1. Test collection.
2. Test execution.

with some calls to pluggy thrown in during both.

From talking with @Zac-HD, I understand there's some appetite for parallelizing collection in pytest itself using threads, perhaps as a consequence of folding pytest-xdist into pytest. This would require making (1) test collection thread safe. I think that parallelized test collection is "merely" a performance improvement, so I'm not proposing it in this issue, though it may be something to keep in mind.

Instead I want to focus on (2); making test execution thread-safe. Here is my understanding of the required work there:

• Make pluggy thread-safe, as hooks will be called from multiple threads.
  • It's possible this work is either minimal or zero, if pluggy is already thread safe.
• Guarantee all (feasible) built-in fixtures are thread-safe.
  • I don't (yet) know pytest internals enough to judge how difficult this will be.
  • Some fixtures, like monkeypatch and capsys, don't seem possible to make thread-safe. Perhaps someone knows an arcanely cursed way to achieve this?
• Make pytest data collection during tests thread-safe.
  • the tracking of the number of xfails, successes, failures, etc.
• Document the thread-safety guarantees, including what is and is not safe.
• Add an opt-in execution model to pytest-xdist which uses threads instead of processes.
  • If we're done our job right up to this point, this should be conceptually simple, though it may require some involved changes to pytest-xdist to support this.

And finally;

• Test the heck out of this.
  • I put a substantial amount of effort into testing thread-safety when migrating Hypothesis: Add threading CI job HypothesisWorks/hypothesis#4489. I would expect to do so again here, though I don't yet know the best way to do so.

Open questions

• If we do leave fixtures like monkeypatch thread-unsafe, how should this interact with pytest-xdist's proposed threading mode? A few options: do nothing; print a warning; error; or skip the test. I don't like half of these!

cc: @ngoldbaum, @lysnikolaou

[RonnyPfannschmidt]

there are multiple missunderstandings here

• pluggy is threadsafe
• nomal fixtures shouldnt have to care about thread-safety unless multi thread fixtures are introduced
• monkeypatch will never be threadsafe, 🤷
• capsys cannot be threadsafe as is as posix file descriptors are not thread isolated
  • multithraded pytest will need stand-in objects with context locals
• collection thread safety is not necessarily required, its ok to leave it as a secondary concern
• xdist should move focus to multi process/multi computer/running in remote environments not gain threads
• sheduler protocols in process can be more powerful than xdist due to direct data access
• execution thread safety will require a more elaborate sheduling mechanism than modifyitems
  • the goal would be to have independent runtestloops with distinct setupstates
• node objects and their stores/stashes need thread safety concerns

so my primary concerns would be detangling setup-state, enabling thread safety in node objects (in particular higher scope nodes)
then would be a follow-up with details like monkeypatch crossover (very hellish, most likely unsafe), log/stdio capture with python level standings & co

fixtures wouldn't need direct thread safety due to explicitly distinct thread safety

but we do need a plan for fixtures that mess with shared state (like modules/monkepatch/fdcapure)

we also need to investigate if subinterpreters will help

[RonnyPfannschmidt]

also additional considerations are needed for fun bits like gui framework eventloops going broke on non-main threads and other curious ecosystem traps

[Liam-DeVoe]

This broadly makes sense, but I wanted to clarify a few things:

xdist should move focus to multi process/multi computer/running in remote environments not gain threads
sheduler protocols in process can be more powerful than xdist due to direct data access

• I had the impression that pytest-xdist gaining an optional threading execution model (pytest -n 8 -> pytest -n 8 --use-threads, pending naming) was a desirable outcome from all parties here. Is that not the case on pytest's end?
• If not, do you see any place in pytest itself to ship such a builtin threaded execution model? Or would you prefer to simply focus on making threaded execution possible to implement, by guaranteeing thread-safety in pytest, and leave implementing it to third-party plugins?
• It seems you want the latter, but possibly adding or changing lifecycle hooks to support the implementation of this by a plugin? (this is my reading of "execution thread safety will require a more elaborate sheduling mechanism than modifyitems").

• execution thread safety will require a more elaborate sheduling mechanism than modifyitems
  • the goal would be to have independent runtestloops with distinct setupstates

Repeating back in my own words: "fixture instances should not be shared across threads; rather, each thread should maintain its own SetupState which pushes/pops/instantiates fixtures for that thread. Fixtures which inherently modify shared state (monkeypatch etc) will need special consideration." This makes sense to me, and I think thread-isolated fixtures is actually what @ngoldbaum recommended initially, and I misread.

[RonnyPfannschmidt]

the core focus of pytest should be making it possible

we also have to consider async task models as possible

xdist has a focus on different processes, execnet and different nodes alltogether

we have had multiple experiments of having a simplefied xdist and i think its going to be something along the lines of splitting it up into scheduling concerns - thread concerns, async concerns - running in remote infra concerns

[Zac-HD]

@RonnyPfannschmidt would you be able to write up your proposal for what Pytest should look like in the free-threaded future? Once we have agreement on that, we can work out how to get there incrementally.

For example, I think most people only use xdist to get multi-core local execution, and we could (imo should eventually) obsolete that with native multithreading in core pytest (with a pull-from-queue efficient scheduler). Then xdist can be multi-computer stuff only, though tbh I think this is usually a mistake and people should do independent runs then combine reports.

But the blocker to all of this seems to be having an agreed list of thread-safety-improving tasks which people can work on!

[RonnyPfannschmidt]

we are probably able to do a rough proposal - but i suspect quite a bit of the details will change as we work towards it and learn

some key elements that come to mind off hand are:

• context aware capture for stdio/logging with the awareness that posix level capture is never thread-safe (this might also help to resolve some long standing developer experience bugs as people keep misunderstanding correct capture configuration for stdio and logging)
• a strategy for monkeypatch - while none may get away with patching some objects , patching the environment, global things and cwd is a potential disaster
• un-tangleing setupstate from nodes/session to enable having more than one per pytest session (this also would be great strides towards a more maintainable fixture system and eventually support for considering type annotations)
• validating if/how the store objects in nodes need to accommodate multi threading
• evaluate if we can do a sensible worker pool version of running collectors
• evaluate scheduling and setup/teardown lineup - in particular wrt expensive higher scope fixtures as

[bluetech]

Regarding monkeypatching etc, I'd say some tests are never going to be thread safe. We'd need a way to mark tests as not thread safe, or more likely the other way around.

[RonnyPfannschmidt]

I think I'd want to be able to explicitly mark fixtures as safe, unsafe or hazard plus propagate implications to unmarked
Fixtures

[Zac-HD]

I agree we'll need that eventually, but I'd start by working on the minimal core, e.g. pytest --capture=no -p no:monkeypatch without use of fixtures. Once we get multiple pytest sessions and node-stores to be threadsafe and tested as such, we can incrementally add all the other stuff users need, and finally the (very, perhaps fundably) nice-to-haves like parallel collection.

I expect that we'll also see some improvements elsewhere in the ecosystem which make this easier, e.g. here's a proposal for contextlib.redirect_stdout() to become thread-aware which we'd be able to use (eventually).

[RonnyPfannschmidt]

Well never be able to use those contextmanagers unless python fundamentally changes io redirection

For thread/logging safety Well have to essentially have thread context aware replacements that include pytest integration

[bluetech]

Well never be able to use those contextmanagers unless python fundamentally changes io redirection

I was also worried about warnings, but it seems that catch_warnings is handled in Python 3.14, so pytest's warnings plugin should work at least.

un-tangleing setupstate from nodes/session to enable having more than one per pytest session

Do I understand correctly that the idea here is that collection is done in a non-parallel phase on the main thread, and the same collection tree is shared among multiple parallel runners? As opposed to xdist which runs collection per worker.

[ngoldbaum]

I was also worried about warnings, but it seems that catch_warnings is handled in Python 3.14, so pytest's warnings plugin should work at least.

And in-practice I haven't seen any thread safety issues due to uses of pytest.warns in 3.14, so any remaining thread safety issues in pytest itself should be hard to trigger, if there are any.

[Liam-DeVoe]

Thanks for the discussion here. Here's a more refined todolist:

• separate SetupState from Session
  • one Session collection-tree should be shared among all threads
  • each thread gets a separate SetupState and therefore a separate instance of each fixture
• audit Stash uses for thread-safey
• make capsys thread-aware, with integrations for the per-thread SetupState

The working goal in my mind is to enable pytest support for the following implementation of a parallel pytest_runtestloop:

from queue import Queue, Empty
from concurrent.futures import ThreadPoolExecutor

n = 2

def pytest_runtestloop(session):
    queue = Queue()
    for item in session.items:
        queue.put(item)

    def worker():
        try:
            item = queue.get_nowait()
        except Empty:
            return

        while item is not None:
            try:
                next_item = queue.get_nowait()
            except Empty:
                next_item = None

            item.config.hook.pytest_runtest_protocol(
                item=item, nextitem=next_item
            )
            item = next_item

    with ThreadPoolExecutor() as executor:
        futures = [executor.submit(worker) for _ in range(n)]
        for future in futures:
            future.result()

    return True

which could be implemented by e.g. pytest-run-parallel or a simplified pytest-xdist as they see fit. (I thought implementing this parallelism in a plugin might require additional pytest hooks, but I think pytest_runtestloop covers it.)

I don't think Quansight or @ngoldbaum necessarily sees it as a requirement that pytest itself ship such a parallel execution model, merely that pytest makes it possible to implement one, though feel free to correct me. As a result, I don't think we yet have to address concerns like "monkeypatch is inherently thread-unsafe", because that can be a caveat/requirement of writing such a parallel pytest_runtestloop. But if we did, then an extension is:

• figure out what to do about inherently-unsafe fixtures like monkeypatch
  • possibly by requiring each fixture to opt-{in, out} to thread safety

[I also forgot to mention in the original post, but #13679 would be wonderful to have merged for pytest-run-parallel!]

---

Do I understand correctly that the idea here is that collection is done in a non-parallel phase on the main thread, and the same collection tree is shared among multiple parallel runners? As opposed to xdist which runs collection per worker.

This is my understanding yes, with a possible eventual goal to make collection itself parallel as an extension to this issue.

[RonnyPfannschmidt]

note that this very naive testloop implementation wrecks havoc wrt sensible reorder to minimize setup/teardown

one would want shared setup/teardowns in "runs" that minimize the need to setup/teardown specific fixtures

aka try to make high scope fixtures sticky to specific treads so we remove the need for setup/teardown cycles

[RonnyPfannschmidt]

oh - and one extra fun consideration - specific event loops will break outside of the main thread (i recall at least QT and gtk)