--cache-clear for pytest xdist

Question

The goal: I need to setup some random value in pytest cache before test collection.

The problem: If I run tests in parallel using pytest-xdist with --cache-clear option master and each worker will clear the cache so I need to make sure that all workers are ready before setting value.

Possible solution:

def pytest_configure(config):
    if (
        hasattr(config, "workerinput")
        and "--cache-clear" in config.invocation_params.args
    ):
        # if it's worker of parallel run with cache clearing,
        # need to wait to make sure that all workers are started.
        # Otherwise, the next started worker will clear cache and create
        # a new organization name
        time.sleep(10)

    name = config.cache.get(CACHE_ORG_KEY, None)
    if not name:
        name = <set random value>
        config.cache.set(CACHE_ORG_KEY, name)

It works fine. I have 10 second sleep and seems it's enough for starting all workers (nodes). All workers are started, all of them clear the cache. The first one sets value to cache and others get it. But I don't like this approach, because there is no guarantee that all workers are started + extra waiting time.

I think about other approaches:

Disable clearing the cache for workers
Check that all workers are started

But I can not figure out how to do it. Any ideas?

UPD #1. Minimal reproducible example

Requirements:

pytest==6.2.5
pytest-xdist==2.5.0

Code:

conftest.py

import time

from test_clear_cache import set_name


def pytest_configure(config):
    # if (
    #     hasattr(config, "workerinput")
    #     and "--cache-clear" in config.invocation_params.args
    # ):
    #     time.sleep(10)
    name = config.cache.get("name", None)
    if not name:
        name = f"name_{time.time_ns()}"
        config.cache.set("name", name)
    set_name(name)

test_clear_cache.py

import sys

NAME = "default"


def set_name(name):
    global NAME
    NAME = name


def test_clear_cache1():
    print(f"Test #1: {NAME}", file=sys.stderr)


def test_clear_cache2():
    print(f"Test #2: {NAME}", file=sys.stderr)


def test_clear_cache3():
    print(f"Test #3: {NAME}", file=sys.stderr)


def test_clear_cache4():
    print(f"Test #4: {NAME}", file=sys.stderr)

Output:

(venv) C:\Users\HP\PycharmProjects\PytestCacheClear>pytest -s -n=4 --cache-clear
========================================================================================================== test session starts ===========================================================================================================
platform win32 -- Python 3.7.8, pytest-6.2.5, py-1.11.0, pluggy-1.0.0
rootdir: C:\Users\HP\PycharmProjects\PytestCacheClear
plugins: forked-1.4.0, xdist-2.5.0
gw0 [4] / gw1 [4] / gw2 [4] / gw3 [4]
Test #4: name_1643377905887805600
Test #3: name_1643377905816748300
Test #2: name_1643377905735875700
Test #1: name_1643377905645880100
....
=========================================================================================================== 4 passed in 0.61s ============================================================================================================

Note: if you uncomment code in conftest.py tests will print the same name.

I may be wrong, but IIRC the cache is cleared before running `configure` hookimpls and the workers don't clear the cache themselves anymore. I also can't reproduce the issue locally; can you add a [mcve] to the question? — hoefling, Jan 28 '22 at 13:24
@hoefling Yes, you are right and the cache is cleared before running `configure`, but workers do clear the cache themselves. As result each worker cleared the cache and set its own value. I want to make sure that all workers are already started and cleared the cache (or disable clearing by them) and the same cache can be set for all of them. Added the example. — Aliaksandr Plekhau, Jan 28 '22 at 14:18
Sorry for the late response; I would split populating cache and module's global var into two parts: write the cache in configure hook and use an autouse fixture that runs on worker level and applies the value from cache to global var. Smth like `@pytest.fixture(autouse=True) def apply_cache(request): set_name(request.config.cache.get('name', None))` and omit calling `set_name()` from configure hook. The issue here is with setting a worker global var from the global hook rather with cache populating. — hoefling, Feb 06 '22 at 13:46
About turning off cache clearing on worker level, you can do that as well via e.g. `def pytest_configure_node(node): node.config.option.cacheclear = False`, but it won't help in your case since it won't affect the wrong value of the module global var. — hoefling, Feb 06 '22 at 13:47

--cache-clear for pytest xdist

0 Answers0