Error when using source in a python Tox ini file

Question

I'm trying to get Tox and Conda to play together well. Mainly because I have a lot of non-python dependencies that need to be installed and it was easy to create Conda distributions. Then I can install everything with a simple conda install.

However, I'm having trouble with activating the conda environment.

[tox]
envlist = py27

[testenv]
whitelist_externals =
    conda
    source
    py.test
setenv =
    PYTHONPATH = {toxinidir}:{toxinidir}/damlarces
install_command =
    python build_env.py --conda-env {toxworkdir}/conda {packages}
commands =
    source activate {toxworkdir}/conda
    py.test --basetemp={envtmpdir}

The python build_env.py --conda-env {toxworkdir}/conda {packages} takes care of creating the environment (if needed), installing the packages, etc. The problem comes at the source activate {toxworkdir}/conda line. I get an ERROR: InvocationError: could not find executable 'source' error. Typing the command directly into the command-line works fine.

For those who are interested. The build_env.py is in this Gist: https://gist.github.com/JudoWill/70450979353fa2d12823 ... Currently its just installing Python dependencies but in its intended environment it will be installing Conda repo's that are not necessarily python libraries. Any thoughts?

I know neither conda nor tox, but since you seem to be able to execute python scripts this way, can't you write a script to wrap that call to `source`, similar to `build_env.py`? — tobias_k, Jul 21 '14 at 12:29
Tried that. Because of the way Tox works, the shell is lost between the `install_command` and the `commands` sections of the run. I've also tried to encapsulate the two `commands` into a single bash-script. This also gives and `InvocationError`. — JudoWill, Jul 21 '14 at 13:23
@alecxe You can see the `build_env.py` code here: https://gist.github.com/JudoWill/70450979353fa2d12823 — JudoWill, Jul 21 '14 at 18:04

score 8 · Answer 1 · answered Jul 21 '14 at 16:03

8

source is not a command. It's a shell builtin. source script.sh causes script.sh to be run inside the current shell. This is necessary for activate, because it modifies the PATH, and you want those modifications to affect the shell itself (normally, when you run a script, it runs in a subshell, which has its own environment that cannot affect the calling shell's environment).

I don't know if tox supports setting environment variables in the commands. If so, you can just use

export PATH={toxworkdir}/conda/bin:$PATH

Otherwise, just use the absolute path to all your commands, like

{toxworkdir}/conda/bin/py.test --basetemp={envtmpdir}

answered Jul 21 '14 at 16:03

asmeurer

86,894
26
169
240

1

Will that reset the path after the program runs? – JudoWill Jul 21 '14 at 18:10
`PATH` will only be set local to tox. Any time you change an environment variable in a program or a script, it only affects the environment for that program, not your shell (the exception is if you have a bash script that you call with `source`). – asmeurer Jul 21 '14 at 20:06
No dice. Now I get an invocation error on the `export` command. `ERROR: InvocationError: could not find executable 'export'` – JudoWill Jul 22 '14 at 12:17
I also tried using the `setenv` section of `tox.ini` to add the path. The problem is that when making a new run the `conda create` hasn't run yet and there's no `conda/bin` directory. – JudoWill Jul 22 '14 at 12:24
That's probably fine. The path doesn't have to exist yet for you to put it on the PATH. – asmeurer Jul 22 '14 at 16:32

score 8 · Accepted Answer · answered Jul 22 '14 at 13:56

After plenty of tinkering I figured out a work-around. Its probably brittle to any changes in Tox but as long as they run tests in the order provided by envlist then it should work.

As suggested by @asmeurer in his answer, the trick is to somehow change the PATH as recognized by tox. But in order to generate the conda/bin I need to run conda create. First, I tried using the export PATH={toxworkdir}/conda/bin:$PATH suggested by asmeurer, but this ran into the same InvocationError issue just with export instead of source.

Next, I tried using the setenv tox section to change the path. This resulted in a chicken & egg problem. I couldn't add the bin directory until I ran the conda create command. Because of the default order of commands in Tox it doesn't seem like I can get the setenv to differ (or re-run) after the install_command.

The work-around I came up with was to create a new env to test against and then share this environment with the subsequent tests.

[tox]
envlist = setup,py27

[testenv]
whitelist_externals =
    conda
    source
    py.test

[testenv:setup]
setenv =
    PYTHONPATH = {toxinidir}:{toxinidir}/damlarces
commands =
    conda config --add channels judowill
    python build_env.py --conda-env {toxworkdir}/conda {packages}

[testenv:py27]
setenv =
    PYTHONPATH = {toxinidir}:{toxinidir}/damlarces
    PATH={toxworkdir}/conda/bin:$PATH
commands =
    {toxworkdir}/conda/bin/py.test --basetemp={envtmpdir}

This works. I'm not sure how difficult it would be to generalize this to a multiple python environment, but it works for me.

Error when using source in a python Tox ini file

2 Answers2