9

I am attempting to setup a project on another laptop than my typical development machine. This project has several pytest-based tests that I have written over the lifetime of the project. When I run

$ pytest -k tests/my_test.py

I get a list of errors from sqlalchemy tests like the following:

_ ERROR collecting env/lib64/python3.5/site-packages/sqlalchemy/testing/suite/test_update_delete.py _
env/lib/python3.5/site-packages/py/_path/local.py:662: in pyimport
    __import__(modname)
env/lib/python3.5/site-packages/sqlalchemy/testing/suite/__init__.py:2: in <module>
    from sqlalchemy.testing.suite.test_cte import *
<frozen importlib._bootstrap>:968: in _find_and_load
    ???
<frozen importlib._bootstrap>:957: in _find_and_load_unlocked
    ???
<frozen importlib._bootstrap>:664: in _load_unlocked
    ???
<frozen importlib._bootstrap>:634: in _load_backward_compatible
    ???
env/lib/python3.5/site-packages/_pytest/assertion/rewrite.py:211: in load_module
    py.builtin.exec_(co, mod.__dict__)
env/lib/python3.5/site-packages/sqlalchemy/testing/suite/test_cte.py:11: in <module>
    class CTETest(fixtures.TablesTest):
env/lib/python3.5/site-packages/sqlalchemy/testing/suite/test_cte.py:99: in CTETest
    @testing.requires.ctes_with_update_delete
E   AttributeError: 'NoneType' object has no attribute 'ctes_with_update_delete'

Why is pytest collecting tests from a dependency? Is it supposed to do this? If not, how can I fix it?

The laptop where I am setting up this project is running Ubuntu 18.04. I created a virtualenv with Python 3.5.6 and ran pip install -r requirements in the virtualenv without any errors, including sqlalchemy and pymssql.

Code-Apprentice
  • 81,660
  • 23
  • 145
  • 268
  • that looks like a missing dependencies issue ... not that its necessarilly "collecting" those tests – Joran Beasley Oct 22 '18 at 20:11
  • 1
    I don't think this is the issue, but are you sure you're using the `-k` flag correctly? https://docs.pytest.org/en/latest/example/markers.html#using-k-expr-to-select-tests-based-on-their-name – Matt Messersmith Oct 22 '18 at 20:15
  • What happens if you run it *without* `-k`? Same thing? – Matt Messersmith Oct 22 '18 at 20:18
  • 1
    @MattMessersmith Removing `-k` seems to do the trick. Tests run (with other errors that I need to resolve). – Code-Apprentice Oct 22 '18 at 20:25
  • 2
    Looks like you have placed the project-specific env in the project root dir (from where you're running `pytest`); if that's the case, you need to ignore that via `norecursedirs` as suggested in the given answer. Same applies to `build` or `.eggs` that tend to emerge at some point and contain tests quite often. – hoefling Oct 22 '18 at 20:35
  • @hoefling Yes, my directory structure includes `$PROJECT_ROOT/env` – Code-Apprentice Oct 22 '18 at 20:39
  • 1
    Well, `pytest` isn't smart enough to recognize that `env` is a virtualenv dir; it treats it like any other directory that may contain tests. If there are packages installed with tests (for example, like `pandas` or `numpy`), `pytest` will find the tests and happily run them. Sometimes it's not even wrong, running tests for all installed packages to check whether the venv is healthy. – hoefling Oct 22 '18 at 20:42
  • However, most of the time you want to exclude those tests from your test suite. You basically have two options: 1. include everything, exclude selected. This is what the `norecursedirs` is responsible for. 2. Exclude everything, include selected. This can be achieved with the `testpaths` option - add `testpaths = tests some/other/test/dir` to your `pytest.ini`, `setup.cfg` or `tox.ini`, whatever you are using. All dirs must be relative to project root dir (it is printed when you run `pytest -v`) and separated by whitespaces. – hoefling Oct 22 '18 at 20:45
  • If you don't like putting stuff into configs, passing the options from command line will also do: `pytest -o norecursedirs=env` or `pytest -o testpaths=tests` etc. – hoefling Oct 22 '18 at 20:51
  • @hoefling Please post an answer. – Code-Apprentice Oct 22 '18 at 20:56

3 Answers3

4

When pytest is invoked, it scans all the child dirs in the project root dir, looking for tests; when you place the project-specific virtual env into project root dir, it will be also scanned without making an exception. This can lead to unwanted tests being included in the test run (for example, when you have dependencies like numpy or pandas that include tests in the distribution).

To circumvent this, pytest offers two configuration options:

  1. norecursedirs - holds directories that will be excluded from scanning. Use this option when you are looking for the pattern "include all, exclude selected". See also wim's answer for the explanation and usage example. By default, norecursedirs is set to .*', 'build', 'dist', 'CVS', '_darcs', '{arch}', '*.egg', so beware that when you override this option, the defaults are gone and you have to add them back.

  2. testpaths - holds directories that should only be considered for the scan, so this is basically the opposite to what norecursedirs is doing. Use this option when looking for the pattern "exclude all, include selected". This option also adds some minor or more significant speedup to the test discovery, depending on what you keep in the project root - most of the sudbirectories won't be traversed at all and the tests run starts sooner.

    Usage: either place the options in the pytest.ini/setup.cfg/tox.ini:

    [tool:pytest]
testpaths = tests othertests doc

    or pass them via --override-ini from command line.

    pytest -o "testpaths=tests othertests doc" ...

Moving the env dir away from the project sources is a matter of choice and depends on what dev environment you have. direnv, pipenv etc. may have different views on where the envs should reside, but personally I don't see any problems in keeping the envs in the project root.

hoefling
  • 59,418
  • 12
  • 147
  • 194
  • If I need to run a single test in the future `pytest -k test_this tests` should avoid recursing down env by specifying a specific directory in the project. – Code-Apprentice Oct 23 '18 at 14:50
3

To answer the question in the title, no.

This may be a symptom of creating your venv and installing packages in a directory that is not ignored. Keep them somewhere ignored, like ~/.venvs/ or whatever.

Alternatively, specifically --ignore=env to exclude the site, or otherwise customize your test discovery in config file:

# content of pytest.ini
[pytest]
norecursedirs = env
wim
  • 338,267
  • 99
  • 616
  • 750
  • by "site-packages" do you mean packages installed on the system outside the virtualenv? either per user or globally? – Code-Apprentice Oct 22 '18 at 20:26
  • I mean the site-packages dir that is available on `sys.path` from within the venv (`env/lib64/python3.5/site-packages/`, according to your tracebacks). Run `python -m site` for details. – wim Oct 22 '18 at 20:31
  • Another alternative, if hoefling's comment is correct, is to move my `env` directory outside of the project folder. – Code-Apprentice Oct 22 '18 at 20:46
  • @Code-Apprentice That was my original suggestion in the answer, wasn't it? ("like `~/.venvs/` or whatever"). Although the accepted answer mentions a good point about the incorrect usage of `-k`, I don't think that's actually the root cause for your problem. – wim Oct 22 '18 at 20:57
  • @wim Yes, your first suggestion is to move my virtual env to an external directory. And removing the `-k` solves my immediate problem. In this project I expect to run complete test files, but if I do need to run individual tests, I will encounter the same problem with the -k option again. – Code-Apprentice Oct 22 '18 at 20:59
  • If I need to run a single test in the future `pytest -k test_this tests` should avoid recursing down env by specifying a specific directory in the project. – Code-Apprentice Oct 23 '18 at 14:50
1

I believe the problem is how you're invoking it. pytest -k tests/my_test.py probably isn't what you want to do: pytest tests/my_test.py probably is, though, as this will invoke the lone test file tests/my_test.py (no recursion will happen: including into your environment site packages). The -k flag is for expressions (so you could run all tests with "http" in the name, for instance).

It doesn't make a lot of sense, but somehow the runner is grabbing some sqlalchemy tests (or they're failing on import, I'm not really sure). Doesn't seem like they should match (even though the runner would recurse into your env directory), but I think it's a bit besides the point. You should be able to fix your issue by simply invoking pytest without the -k option.

P.S. I can't reproduce the behavior you're seeing on my windows box.

HTH.

Matt Messersmith
  • 12,939
  • 6
  • 51
  • 52
  • I don't think the test runner even makes it to the matching phase because it errors out on the "collecting tests to match against" phase. – Code-Apprentice Oct 22 '18 at 21:00