How to let pytest discover and run doctests in installed modules?

Question

I am adding unit tests and to a kind of "legacy" Python package. Some of the modules contain their own doctests embedded in docstrings. My goal is to run both those doctests and new, dedicated unit tests.

Following this Q&A ("How to make py.test run doctests as well as normal tests directory?") I'm using the --doctest-modules option to pytest. When running from the source repository, pytest indeed discovers the embedded doctests from Python modules under the src directory.

However, my goal is to test that the source distribution builds and installs at all, and then test everything against the installed package. To do this I'm using tox which automate the process of building a sdist (source distribution) tarball, installing it in a virtual environment, and running the tests against the installed version. To ensure that it is the installed version, rather than the one in the source repository, that is imported by the tests, I follow the suggestion in this article and the repository looks like this now:

repo/
  src/
    my_package/
      __init__.py
      module_a.py
      module_b.py
      ...
  tests/
    test_this.py
    test_that.py
  requirements.txt
  setup.py
  tox.ini

(The test scripts under tests import the package as in import my_package, which hits the installed version, because the repository layout makes sure that the src/my_package directory is out of the module search paths.)

And in the tox configuration file, the relevant sections looks like

[tox]
envlist = py27,py36,coverage-report

[testenv]
deps =
  -rrequirements.txt
commands =
  coverage run -p -m pytest --

[pytest]                                                                        
addopts = --doctest-modules 

So far, the tests run fine, and the doctests are picked up -- from the modules under src/my_package, rather than from the package installed in tox virtual environments.

My questions related to this set-up is as follows:

• Is this actually a concern? tox seems to ensure that what you install is what you have in the source repository, but does it?
• How can I instruct pytest to actually run doctests from the installed modules in a sort of clean way? I can think of a few solutions such as building a dedicated documentation tree in a doc directory and let pytest find the doctests in there with the --doctest-glob option. But is there a method to do this without building the docs first?

Answer 1

I found the answer to my own question.

To quote pytest issue #2042:

currently doctest-modules is fundamentally incompatible with testing against a installed package due to module search in checkout vs module usage from site-packages

So as of now the solution does not exist.

Answer 2

pytest collects tests from the current directory (unless you instruct it otherwise passing an explicit directory). Add the installation directory using tox substitutions in tox.ini. I.e., either pass the directory:

[testenv]
deps =
  -rrequirements.txt
commands =
  coverage run -p -m pytest {envsitepackagesdir}/my_package

or change directory:

[testenv]
changedir = {envsitepackagesdir}/my_package
deps =
  -rrequirements.txt
commands =
  coverage run -p -m pytest --

Answer 3

This is how I solved the import mismatch:

$ pytest tests/ --doctest-modules --pyargs myrootpkg <other args>

The problem here is that once you start specifying source paths explicitly (in this case via --pyargs), you have to specify all other source paths as well (tests/ in this example) as pytest will stop scanning the rootdir. This shouldn't be an issue when using the src layout though, as the tests aren't usually scattered around the repository.