pytest-xdist with module-scoped fixture

Question

How many times will the fixture run for:

@pytest.fixture(scope="module")
def foo():
   return True

def test1(foo):
   pass

def test2(foo):
   pass

Answer 1

It will run once per xdist worker thread.

This is not what you'd expect as a new user, but the explanation is that each xdist worker process is actually running one complete distinct session, only with a subset of the test cases in each.

This misassupmtion that one pytest invocation equals one test session hit me as well today, only worse, because I had a parameterized cpu and memory heavy module-scoped fixture (running the tested binary as a subprocess on many different input files), used in many tests cases, so it was run once per testcase per file per worker and pegged my cpu for minutes and then made me run out of swap.

My solution was to use pytest -n auto --dist=loadscope, to run all tests that used that specific fixture in sequence, and still do other useful work in parallel.

There are useful workarounds for some use cases linked to in this issue from 2018: https://github.com/pytest-dev/pytest-xdist/issues/271

the discussion itself is much older, from 2013 in the pytest repo. They ultimately said it wasnt their problem, as sessions work as expected when not using the xdist add-on: https://github.com/pytest-dev/pytest/issues/252