How to access a shared resource with pytest-xdist?

Question

I want to access a list that contains all the account credentials to provide them to each separate thread in pytest-xdist. How do I achieve that? As far as I know, pytest-dist, for each thread started, it is a separate session of tests and they initialize each python module separately in their memory. I have a example of code snippet as follows

import sys
import pytest
import threading
import time

lock = threading.Lock()

i = 0
lis = []
lis.append(1)
lis.append(2)


class TestAAA(object):

    def test_function1(self, fixture_function_feature1):
        global i, lock
        with lock:
            print "Lock Acquired"
            time.sleep(2)
            print >> sys.stderr, 'test_: '+str(lis[i])

Command executed:

pytest -sv -n 2 --count=5

Output:

test_: 1
test_: 1

Output expected:

test_: 1
test_: 2

I also tried using a shared resource mentioned at https://github.com/pytest-dev/pytest/issues/1402#issuecomment-186299177

 import pytest

    def pytest_configure(config):        
        if is_master(config):
            config.shared_directory = tempdir.mktemp()

    def pytest_unconfigure(config):        
        if is_master(config):
            shutil.rmtree(config.shared_directory)


    def pytest_configure_node(self, node):
        """xdist hook"""
        node.slaveinput['shared_dir'] = node.config.shared_directory


    @pytest.fixture
    def shared_directory(request):
        if is_master(request.config):
            return request.config.shared_directory
        else:
            return request.config.slaveinput['shared_dir']


    def is_master(config):
        """True if the code running the given pytest.config object is running in a xdist master
        node or not running xdist at all.
        """
        return not hasattr(config, 'slaveinput')

    def test_shared_dir(shared_directory):
        print >> sys.stderr, 'master logs dir: '+str(shared_directory)

Running them with pytest-xdist and without, both gives out error

Command: pytest -sv test_parallel_share.py

Output:

―――――――――――――――――――――――――――――――――――――――――――――― ERROR at setup of test_shared_dir ―――――――――――――――――――――――――――――――――――――――――――――――

request = <SubRequest 'shared_directory' for <Function 'test_shared_dir'>>

    @pytest.fixture
    def shared_directory(request):
        if is_master(request.config):
>           return request.config.shared_directory
E           AttributeError: 'Config' object has no attribute 'shared_directory'

test_parallel_share.py:21: AttributeError
                                                                                                                 100% ██████████

Results (0.05s):
       1 error

Command: pytest -n 2 -sv test_parallel_share.py

Output:

scheduling tests via LoadScheduling


―――――――――――――――――――――――――――――――――――――――――――――― ERROR at setup of test_shared_dir ―――――――――――――――――――――――――――――――――――――――――――――――

request = <SubRequest 'shared_directory' for <Function 'test_shared_dir'>>

    @pytest.fixture
    def shared_directory(request):
        if is_master(request.config):
            return request.config.shared_directory
        else:
>           return request.config.slaveinput['shared_dir']
E           KeyError: 'shared_dir'

test_parallel_share.py:23: KeyError
                                                                                                                 100% ██████████

Results (0.55s):
       1 error

Right now, it looks like the [XY problem](https://meta.stackexchange.com/questions/66377/what-is-the-xy-problem) to me, but maybe I'm wrong - what is the purpose of the credentials list? Do you just want to run the same test with different credentials in parallel? — hoefling, Jul 23 '18 at 19:32
Yes, that is correct. Same test with different credentials so that it can sign in into different accounts. — Milin, Jul 24 '18 at 22:08
They are parameterized, the logs need to be written to a common dir by all of them which has a name with timestamp in it. — Milin, Jul 25 '18 at 23:08

msudder · Answer 1 · 2022-08-20T14:39:00.560

I think there are a number of ways to address your question. config.option content is shared between master and workers. A special hook can customize runtime info...

There is no point in testing for is_master in fixtures, accepting its worth is for the case where xdist is not present (no -n). When xdist is in play, fixtures are not run in master, only hooks. Fixtures are only executed inside the child process inside normal protocols wrapped in pytest_runtestloop.

The custom key in slave_input you dereference needs to be set from master in the hook that prepares the slave. Look in the pytest-xdist newhooks.py and you will find something like a home for setting that key in node.workerinput. Look at pytest_configure_node(self, node: WorkerController). You should write one of these and set node.workerinput['shared_dir'] there, the slaves will find common data you want to distribute.

def pytest_configure_node(self, node: WorkerController):
    node.workerinput['shared_dir'] = node.config.option.shared_dir

How to access a shared resource with pytest-xdist?

I also tried using a shared resource mentioned at https://github.com/pytest-dev/pytest/issues/1402#issuecomment-186299177

1 Answers1