Running pytest tests against multiple backends?

Question

I've built a series of tests (using pytest) for a codebase interacting with the Github API (both making calls to it, and receiving webhooks).

Currently, these tests run against a semi-realistic mock of github: calls to github are intercepted through Sentry's Responses and run through a fake github/git implementation (of the bits I need), which can also be interacted with directly from the tests cases. Any webhook which needs to be triggered uses Werkzeug's test client to call back into the WSGI application used as webhook endpoint.

This works nicely, fast (enough) and is an excellent default, but I'd like the option to run these same tests against github itself, and that's where I'm stumped: I need to switch out the current implementations of the systems under test (direct "library" access to the codebase & mock github) with different ones (API access to "externally" run codebase & actual github), and I'm not quite sure how to proceed.

I attempted to use pytest_generate_tests to switch out the fixture implementations (of the codebase & github) via a plugin but I don't quite know if that would even work, and so far my attempts to load a local file as plugin in pytest via pytest -p <package_name> have not been met with much success.

I'd like to know if I'm heading in the right direction, and in that case if anyone can help with using "local" plugins (not installed via setuptools and not conftest.py-based) in pytest.

Not sure if that has any relevance, but I'm using pytest 3.6.0 running on CPython 3.6, requests 2.18.4, responses 0.9.0 and Werkzeug 0.14.1.

You can create a [parametrized fixture](https://docs.pytest.org/en/latest/fixture.html#fixture-parametrize) that returns different backend implementations. Once injected in a test, it will create different instances of the same test that run with different backends. If this is not what you want, consider creating an [mcve](https://stackoverflow.com/help/mcve). — hoefling, Jun 04 '18 at 11:41

score 1 · Accepted Answer · answered Jun 04 '18 at 18:09

There are several ways to approach this. The one I would go for it by default run your mocked tests and then when a command line flag is present test against both the mock and the real version.

So first the easier part, adding a command line option:

def pytest_addoption(parser):
    parser.addoption('--github', action='store_true',
                 help='Also test against real github')

Now this is available via the pytestconfig fixture as pytestconfig.getoption('github'), often also indirectly available, e.g. via the request fixture as request.config.getoption('github').

Now you need to use this parametrize any test which needs to interact with the github API so that they get run both with the mock and with the real instance. Without knowing your code it sounds like a good point would be the Werkzeug client: make this into a fixture and then it can be parameterized to return both a real client or the test client you mention:

@pytest.fixture
def werkzeug_client(werkzeug_client_type):
    if werkzeug_client_type == 'real':
        return create_the_real_client()
    else:
        return create_the_mock_client()

def pytest_generate_tests(metafunc):
    if 'werkzeug_client_type' in metafunc.fixturenames:
        types = ['mock']
        if metafunc.config.getoption('github'):
            types.append('real')
        metafunc.parametrize('werkzeug_client_type', types)

Now if you write your test as:

def test_foo(werkzeug_client):
     assert werkzeug_client.whatever()

You will get one test normally and two tests when invoked with pytest --github.

(Be aware hooks must be in conftest.py files while fixtures can be anywhere. Be extra aware that the pytest_addoption hook should really only be used in the toplevel conftest file to avoid you from confusion about when the hook is used by pytest and when not. So you should put all this code in a toplevel conftest.py file really.)

Accepting this answer because it's answering the direction question (with the belief that it is correct), and it could be useful for other visitors. My fixtures set is a bit too complex for such a trivial conditional but after some time on the pytest IRC and help from Ronny Pfannschmidt (I think) I managed to set up "exclusive plugins" such that the local/mock implementation of various fixtures is used by default, but I can invoke pytest with -p remote_plugin to avoid loading the mock fixtures and load "real" fixtures instead. — Masklinn, Jun 05 '18 at 08:22

Running pytest tests against multiple backends?

1 Answers1

Linked