4

I'm using tox to prepare venv and run unit tests and my application needs openopt library which in turn imports numpy.distutils.core in its setup.py.

No matter how I order numpy and openopt in my requirements.txt I can't ensure numpy is installed before setup.py from openopt is executed and exit with ImportError: No module named numpy.distutils.core

How can I fix that? For development I can add numpy to requirements.txt, run tox, add openopt and run tox again but it's not production-ready setup.

Oliver Bestwalter
  • 5,219
  • 33
  • 47
Chulup
  • 460
  • 6
  • 10
  • 2
    Found that [pip won't implement build time dependencies](https://github.com/pypa/pip/issues/2381). Now is there a workaround for that issue? – Chulup Apr 12 '16 at 13:41
  • Have you considered a docker container (using Kitematic), which has helped me in similar situations. – roadrunner66 Apr 12 '16 at 14:50

3 Answers3

3

UPDATE There is an issue in the tox project that might be implemented that would add functionality to deal with these kinds of problems in a more "official" way. Discussion is here: Add an option to run commands after virtualenv creation but before other steps

UPDATE (a bit more background): The main problem is that it is a BadThing(TM) to assume that some other package is installed already in setup.py. these kinds of problems fall into the area of bootstrapping and they can be hellish to handle properly, but usually this is possible with some extra effort. If you really need a different package at setup time, you can look into setup_requires and some additional magic (have a look e.g. at setuptools_scm for inspiration). In the worst case and if the package is not to complicated, you can make it part of your package (which comes with its own problems though, like keeping it up to date and possible licensing conflicts).

Original answer:

If you use requirements.txt already, an easy (but admittedly ugly) solution would be:

  1. create two (or more) requirements files (e.g. requirements-0.txt and requirements-1.txt (hopefully with better names)).
  2. sort the packages by dependency into those files
  3. use commands instead of deps to install them in the right order

.e.g.

[testenv]
deps = 
    pytest
    # whatever else where order does not matter

commands =
    pip install -r {toxinidir}/requirements-0.txt
    pip install -r {toxinidir}/requirements-1.txt
    # ... and more if needed

    # now do your actual testing ...
    pytest tests/unit

... or if you want to keep it even simpler, just stick the package that is being imported in setup.py of another package right in front of your single requirements.txt

[...]
commands =
    pip install <package that needs to be installed first (e.g. numpy)>
    pip install -r {toxinidir}/requirements.txt        
    pytest tests/unit
Oliver Bestwalter
  • 5,219
  • 33
  • 47
  • 2
    This won't work because tox installs the package before executing the `commands` section. And if the dependency is mentioned/is required in `setup.py` the package would not install at all inside the tox environment. – jadelord Jul 04 '17 at 16:27
  • You're correct for the scenario you state (having conflicting dependencies in setup.py), but this is not what the question was about. – Oliver Bestwalter Jul 04 '17 at 20:57
  • So just to be clear, this would require the OP to remove `openopt` from his package's `setup.py`. Instead should be installed explicitly. – jadelord Sep 01 '17 at 09:31
  • @jadelord - OP did not have openopt in their setup.py. openopt imported numpy in its setup.py. OP has a requirements.txt conatining both numpy and openopt and openopt was always installed first. This was the cause of the problem and you can only make that work by making sure numpy is installed before openopt is installed. I realized though that using was confusing because the nasty package is openopt in that case. I will clarify my answer with the concrete packages. – Oliver Bestwalter Sep 01 '17 at 13:35
0

It's documented in https://testrun.org/tox/latest/example/basic.html#depending-on-requirements-txt

deps = -rrequirements.txt

According to common practice on github, common trick is:

deps =
    setuptools
    -r{toxinidir}/requirements.txt
Dima Tisnek
  • 11,241
  • 4
  • 68
  • 120
  • 2
    This does not solve the problem, as those would be called together as `pip install setuptools -r./requirements.txt`. See https://travis-ci.org/obestwalter/tox-reproducers/jobs/212139134 and https://github.com/obestwalter/tox-reproducers/blob/cf89b7cd445664a01c5d73fb14bd16b01d0a5427/README.md – Oliver Bestwalter Mar 17 '17 at 14:43
0

I have a generic way to bootstrap build-time dependencies in setup.py. You can use this even if you are not using tox. For this case, add the following snippet to the top of the setup.py script.

from setuptools.dist import Distribution

# Bootstrapping dependencies required for the setup
Distribution(dict(setup_requires=['numpy']))

Warning: This will install numpy using easy_install. Installing numpy with this method is somewhat tricky.

jadelord
  • 1,511
  • 14
  • 19
  • Note: This method works only with setuptools <= 36.6.0. With [PEP 518](https://www.python.org/dev/peps/pep-0518/) getting implemented with pip 10.0.0 we could have a cleaner solution. – jadelord Nov 20 '17 at 11:07