Adding requirements to "run" or "host" fails conda-build

Question

I'm using conda-build to build a conda package from python source code, and I'm getting the following error whenever I add something to the "run" or "host" subsections of the "requirements" section in meta.yaml:

Tests failed for my_package-0.1.0-0.tar.bz2 - moving package to /home/ec2-user/anaconda3/conda-bld/broken

Removing the "run" and "host" subsections makes the test run fine - the built tar.bz2 file is installed without issue, and the import succeeds. Adding a "build" subsection works fine. Running with the --debug flag didn't add any useful information. How can I debug this??

this is my meta.yaml:

package:
  name: my_package
  version: 0.1.0

source:
  path: ..

build:
  script: "python setup.py install --single-version-externally-managed --record=record.txt --verbose"

requirements:
  # removing this subsection makes everything work
  run:
    - python

this is my run_test.sh (using a run_test.py instead produces same result):

echo 'test is running' > /tmp/test_ran.txt    
python -c "import my_package; print('Success!')" >> /tmp/test_ran.txt

this is my setup.py:

from setuptools import find_packages, setup

setup(
    name='my_package',
    version='0.1.0',

    packages=find_packages()
)

The meta.yaml and run_test.sh scripts are in a conda.recipe subfolder under the project root.

For some reason, the test script isn't even being run (the file /tmp/test_ran.txt is not created)! As mentioned above, removing the "run" subsection makes the test run just fine, including the import.

Thanks in advance. Can anyone please help? Going nuts over this...

Stuart Berg · Answer 1 · 2020-08-09T14:07:30.203

1

When you remove the host and run sections, you are omitting python from the environments used during the build/test process. Therefore, the only remaining python on your PATH is the system interpreter (or maybe some other python that happens to be available on your PATH).

That is, you are accidentally using the system python during the build phase and again during the test phase:

Your build command (python setup.py install ...) will use the system interpreter if no python exists in the host or build environments.
Similarly, your run_script.sh script will also use the system interpreter, but it passes -- your build step installed the package into the system interpreter!

OK, so you definitely need to keep the host and run sections in meta.yaml, and then figure out how to get the tests to pass in the correct environment. But it's hard to know why your tests are failing, since you didn't post any info about why the tests are failing.

Instead of run_test.sh, a minimal sanity check would be to add some test imports to meta.yaml:

package:
  name: my_package
  version: 0.1.0

source:
  path: ..

build:
  # BTW, I recommend using {{ PYTHON }} here -- avoid the system interpreter!
  script: "{{ PYTHON }} setup.py install --single-version-externally-managed --record=record.txt --verbose"

requirements:
  host:
    - python
  run:
    - python

# Add this section!
test:
  imports:
    - my_package

Does that pass, at least? If not, inspect the temporary test environment that conda-build was using at the time of the failure.

edited Aug 09 '20 at 14:07

answered Aug 08 '20 at 18:36

Stuart Berg

17,026
12
67
99

thanks Stuart. The above also does not pass. Looking at the conda-bld/broken dir, I see that a tar.bz2 file is created, and I can `conda install` this tar.bz2 and then import the python package successfully. What else can I check? The conda-build output shows no further information on why tests fail. – OronNavon Aug 09 '20 at 06:57
I don't have any brilliant ideas. Maybe try adding some echo statements to `run_test.sh`, just in case it IS being executed, but somehow exiting early? Use `set -x` so you can see how far it gets. Also, inspect the scripts in `/home/ec2-user/anaconda3/conda-bld/my_package_*/test_tmp` to see if they look correct. There should be three scripts there: `conda_test_env_vars.sh`, `conda_test_runner.sh`, and `run_test.sh`. – Stuart Berg Aug 09 '20 at 14:15
Also, just a sanity check: Your machine has `bash` installed, right? And it's located at `/bin/bash`? – Stuart Berg Aug 09 '20 at 14:18
thanks for the help Stuart. Still working on it, will update if I find anything. – OronNavon Aug 12 '20 at 07:05
Found and fixed. Strange issue related to conda activation of Keras. I'm not sure if it's particular to my setup, I'll write it up in case anyone else has the same. Thanks again for your help Stuart! – OronNavon Aug 12 '20 at 08:31

score 1 · Accepted Answer · edited Aug 12 '20 at 08:43

I'm not sure if this was due to my particular setup, but in case someone else has the same issue:

The problem was an error being thrown in one of the nested scripts run by conda-build. Specifically conda creates and activates conda environments in which to run the tests, and the activation of the conda env on my system included activation of Keras for some reason. From the xtrace, just before conda-build reported tests failed:

++++ . /home/my_user/anaconda3/etc/conda/activate.d/keras_activate.sh
++++++ python /home/my_user/anaconda3/etc/keras/load_config.py
+++++ KERAS_BACKEND=tensorflow
+++++ python -c 'import keras'
+++++ test true
+++++ export KERAS_BACKEND=theano
+++++ KERAS_BACKEND=theano
+++++ python -c 'import keras'

Since keras was not specified anywhere in my meta.yaml, the import failed, causing conda-build to exit (with the unhelpful error of "tests failed"). Adding keras under the "test.requires" section in meta.yaml fixed the issue. I don't know why conda-build was trying to activate keras, it might be particular to my setup.

Adding requirements to "run" or "host" fails conda-build

2 Answers2