3

I have a conda environment, myenv. I realized that the Python interpreter in this environment tries to import packages from /Users/me/.local/lib/python3.8/site-packages before /usr/local/anaconda3/envs/myenv/lib/python3.8/site-packages. I would have expected all the environment-specific paths to be appended to the beginning of sys.path. Is this expected behavior? In this case, I want to import conda-installed versions of numpy, scipy and numexpr since they use the Intel MKL backend.

(myenv) me$ which python
/usr/local/anaconda3/envs/myenv/bin/python
(myenv) me$ python
Python 3.8.5 (default, Sep  4 2020, 02:22:02) 
[Clang 10.0.0 ] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> np.__file__
>>> '/Users/me/.local/lib/python3.8/site-packages/numpy/__init__.py'
>>> import sys
>>> # /Users/me/dir* refer to directories that contain local modules I've pip installed
>>> print("\n".join(sys.path))

/usr/local/anaconda3/envs/myenv/lib/python38.zip
/usr/local/anaconda3/envs/myenv/lib/python3.8
/usr/local/anaconda3/envs/myenv/lib/python3.8/lib-dynload
/Users/me/.local/lib/python3.8/site-packages
/Users/me/dir1
/Users/me/dir2
/Users/me/dir3
/usr/local/anaconda3/envs/myenv/lib/python3.8/site-packages

Other things that might be relevant:

  • I use conda and pyenv side-by-side so have conda's auto_activate_base setting set to false (as suggested here).
  • The Python interpreter declares that it's version 3.8.5, but conda info lists python version as 3.8.3.final.0.

I'm using conda version 4.9.2 on a MacOS Version 11.2.2.

sjplural
  • 663
  • 1
  • 8
  • 11
  • 1
    It would be nice to see what is on PATH and if PYTHONPATH is set, or maybe `which -a python` output. Conda really doesn't play well with packages installed with `pip install --user` which is I think how the `~/.local/lib/...` packages get there - probably need to uninstall those. Also, is `.bash_profile` set with default `conda init`? – merv Mar 22 '21 at 22:56
  • Thanks so much for pointing out the issues with using `pip install --user` alongside conda environments! This helped me resolve the issue (see answer below). In case it's helpful for others to see if they're having the same issue, `which python` outputs `/usr/local/anaconda3/envs/myenv/bin/python` and yes, `.bash_profile` was set with default `conda init`. – sjplural Mar 23 '21 at 23:53

1 Answers1

1

Thanks to @merv's comment, I realized that the issue was that pip's user-site is prioritized above the conda environment's package directory. I resolved this as @merv suggested: by uninstalling all the packages in pip's user-site.

me$ pyenv shell 3.8.1
me$ pip freeze --user | grep -Eo "[[:alnum:]\-]+==[[:digit:]]{1,2}\.[[:digit:]]{1,2}(\.[[:digit:]]{1,2})?" | xargs pip uninstall -y
me$ conda activate myenv
(myenv) me$ python
Python 3.8.5 (default, Sep  4 2020, 02:22:02) 
[Clang 10.0.0 ] :: Anaconda, Inc. on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import numpy as np
>>> np.__file__
/usr/local/anaconda3/envs/myenv/lib/python3.8/site-packages/numpy/__init__.py
sjplural
  • 663
  • 1
  • 8
  • 11
  • 1
    For reference, [here is official documentation](https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#pip-in-env) that warns against using the `--user` flag, and [here is a blog post](https://www.anaconda.com/blog/using-pip-in-a-conda-environment) from one of the core devs discussing some of the background to those recommendations. – merv Mar 24 '21 at 02:21