Recommended way to execute tests within Docker container

Question

I would like to run set of specific tests within Docker container and not sure how to tackle this. Tests I want to perform are security-related, like create user(s), manage GPG keys for them and similar - which I am reluctant to run on PC running the tests.

I tried pytest-xdist/socketserver combo and also copying tests into running Docker container and use pytest-json-report to get result(s) as json saved to a volume shared with the host, but not sure this approach is good.

For now, I would settle with all tests (without mark or similar other features) are executed "remotely" (in Docker) and results are presented like everything is ran on local PC.

Don't mind writing a specific plugin, but not sure if this is a good way: do I have to make sure than, my plugin is loaded before say, pytest-xdist (or some others)? Additionally, if I use say, pytest_sessionstart in my conftest.py to build Docker image that I can then target with xdist; but my tests have also some "dependency" that I have to put within conftest.py - I cant use same conftest.py within container and in my PC running the test.

Thank you in advance

Answer 1

In case anyone else maybe have similar need, I will share what I did.

First of all, there is already an excellent pytest-json-report to export JSON results. However, I made simpler and with less functionality plugin that uses pytest_report_to_serializable directly:

import json
from socket import gethostname


def pytest_addoption(parser, pluginmanager):
    parser.addoption(
        '--report-file', default='%s.json' % gethostname(), help='path to JSON report'
    )


def pytest_configure(config):
    plugin = JsonTestsExporter(config=config)
    config._json_report = plugin
    config.pluginmanager.register(plugin)


def pytest_unconfigure(config):
    plugin = getattr(config, '_json_report', None)
    if plugin is not None:
        del config._json_report
        config.pluginmanager.unregister(plugin)

    print('Report saved in: %s' % config.getoption('--report-file'))


class JsonTestsExporter(object):

    def __init__(self, config):
        self._config = config
        self._export_data = {'collected': 0, 'results': []}

    def pytest_report_collectionfinish(self, config, start_path, startdir, items):
        self._export_data['collected'] = len(items)

    def pytest_runtest_logreport(self, report):
        data = self._config.hook.pytest_report_to_serializable(
            config=self._config, report=report
        )
        self._export_data['results'].append(data)

    def pytest_sessionfinish(self, session):
        report_file = self._config.getoption('--report-file')
        with open(report_file, 'w+') as fd:
            fd.write(json.dumps(self._export_data))

Reason beyond this is that I wanted results also imported using pytest_report_from_serializable.

Simplified Dockerfile:

FROM debian:buster-slim AS builder

COPY [ "requirements.txt", "run.py", "/artifacts/" ]
COPY [ "json_tests_exporter", "/artifacts/json_tests_exporter/" ]

RUN apt-get update\
# install necesssary packages
 && apt-get install --no-install-recommends -y python3-pip python3-setuptools\
# build json_tests_exporter *.whl
 && pip3 install wheel\
 && sh -c 'cd /artifacts/json_tests_exporter && python3 setup.py bdist_wheel'

FROM debian:buster-slim

ARG USER_UID=1000
ARG USER_GID=${USER_UID}

COPY --from=builder --chown=${USER_UID}:${USER_GID} /artifacts /artifacts

RUN apt-get update\
# install necesssary packages
 && apt-get install --no-install-recommends -y wget gpg openssl python3-pip\
# create user to perform tests
 && groupadd -g ${USER_GID} pytest\
 && adduser --disabled-password --gecos "" --uid ${USER_UID} --gid ${USER_GID} pytest\
# copy/install entrypoint script and preserver permissions
 && cp -p /artifacts/run.py /usr/local/bin/run.py\
# install required Python libraries
 && su pytest -c "pip3 install -r /artifacts/requirements.txt"\
 && su pytest -c "pip3 install /artifacts/json_tests_exporter/dist/*.whl"\
# make folder for tests and results
 && su pytest -c "mkdir -p /home/pytest/tests /home/pytest/results"

VOLUME [ "/home/pytest/tests", "/home/pytest/results" ]

USER pytest

WORKDIR /home/pytest/tests

ENTRYPOINT [ "/usr/local/bin/run.py" ]

JSON exporter plugin is located in same folder as Dockerfile

run.py is as simple as:

#!/usr/bin/python3

import pytest
import sys
from socket import gethostname


def main():
    if 1 == len(sys.argv):
        # use default arguments
        args = [
            '--report-file=/home/pytest/results/%s.json' % gethostname(),
            '-qvs',
            '/home/pytest/tests'
            ]
    else:
        # caller passed custom arguments
        args = sys.argv[1:]

    try:        
        res = pytest.main(args)
    except Exception as e:
        print(e)
        res = 1

    return res


if __name__ == "__main__":
    sys.exit(main())

requirements.txt only contains:

python-gnupg==0.4.4
pytest>=7.1.2

So basically, I can run everything with:

docker build -t pytest-runner ./tests/docker/pytest_runner
docker run --rm -it -v $(pwd)/tests/results:/home/pytest/results -v $(pwd)/tests/fixtures:/home/pytest/tests pytest-runner

Last two lines I made programatically run from Python in pytest_sessionstart(session) hook using Docker API.