5

In a Conda environment (base here), I'm surprised by the order of directories in the Python path:

python -c "import sys; print(sys.path)"
['', 
'/export/projects/III-data/wcmp_bioinformatics/db291g/miniconda3/lib/python37.zip',
'/export/projects/III-data/wcmp_bioinformatics/db291g/miniconda3/lib/python3.7',
'/export/projects/III-data/wcmp_bioinformatics/db291g/miniconda3/lib/python3.7/lib-dynload',
'/export/home/db291g/.local/lib/python3.7/site-packages',
'/export/projects/III-data/wcmp_bioinformatics/db291g/miniconda3/lib/python3.7/site-packages']

As you can see, my local non-Conda path:

/export/home/db291g/.local/lib/python3.7/site-packages

comes before the Conda counterpart:

/export/projects/III-data/wcmp_bioinformatics/db291g/miniconda3/lib/python3.7/site-packages

This means that Python packages installed in miniconda3/lib/python3.7/site-packages will be ignored if they are also found in .local/lib/python3.7/site-packages. In fact, in .local/lib/python3.7/site-packages I have numpy v1.20, but in the Conda environment I need v1.19, which is correctly installed, but superseded by v1.20. This seems to defeat the point of using Conda.

Is there something wrong with my configuration or am I missing something here?

Some info:

which python
/export/projects/III-data/wcmp_bioinformatics/db291g/miniconda3/bin/python

python -V
Python 3.7.12

which conda
/export/projects/III-data/wcmp_bioinformatics/db291g/miniconda3/bin/conda

conda --version
conda 4.11.0
merv
  • 67,214
  • 13
  • 180
  • 245
dariober
  • 8,240
  • 3
  • 30
  • 47

1 Answers1

8

This is expected behavior (see PEP 370) and partially why Anaconda recommended against user-level package installations.

The site module is responsible for setting the sys.path when Python is initializing. The code in site.py specifically appends the user site prior to appending the prefix site, which is what leads to this prioritization. The motivation according to PEP 370 is that users would have a Python installed at system-level, but want to prioritize packages they install at the user level, hence the user site should load prior to the prefix site.

Options

There are several options for avoiding the user-level site-packages from getting loaded.

1: Environment variable

The environment variable PYTHONNOUSERSITE will toggle loading of user-level site-packages. Namely,

PYTHONNOUSERSITE=1 python -c "import sys; print(sys.path)"

2: Python -s flag

Alternatively, the Python binary has an -s argument to specifically disable user-level site packages.

python -s -c "import sys; print(sys.path)"

3: Remove (and avoid future) user-level installs

The Conda recommendation is to avoid pip install --user altogether, which would be interpreted that one should remove the ~/.local/lib/python* folders from your system.

4: Automated Conda environment variable

Conda Forge package

The Conda Forge package conda-ecosystem-user-package-isolation will automatically set PYTHONNOUSERSITE=1 during environment activation.

If you would like all environments to have such isolation by default, then consider adding this to the create_default_packages configuration list:

conda config --add create_default_packages conda-ecosystem-user-package-isolation

Note that this package also sets R_LIBS_USER="-", isolating any R environments from user-level packages.

Alternative packages

If you want a more granular option, I have also created separate packages that set just the PYTHONNOUSERSITE=1 and PYTHONPATH="" environment variables, which can be installed with:

## set PYTHONNOUSERSITE=1
conda install merv::envvar-pythonnousersite-true

## clear PYTHONPATH
conda install merv::envvar-pythonpath-null
merv
  • 67,214
  • 13
  • 180
  • 245
  • Thanks for the thorough answer. Your option 3 (remove `~/.local/lib/python*`) perhaps is too aggressive. It seems to me that one should routinely start with `conda activate my-env && export PYTHONNOUSERSITE=1` so that either you import what you have installed in the env or you fail with a meaningful import error. The current behavior is quite tricky...! – dariober Feb 02 '22 at 21:07
  • 2
    @dariober added another option: automatic environment variable setting via a Conda package. It was kinda a proof-of-concept I've been wanting to try out for awhile. Might be useful for those who still want the user site and want a quick solution to insulate their Conda environment. – merv Feb 03 '22 at 03:45
  • This looked like the answer I wanted, but it doesn't work for me. When I use the -s option for python (version 3.9.13), the path still has some local directories. I cannot find the source of those local directories (no conda.pth in my env, nothing obvious anywhere in the env directory tree, etc). – Michael Tiemann Nov 28 '22 at 21:27
  • 1
    Following up on my own comment, I found this file that's likely the source of grief: /Users/michael/miniconda3/envs/itr_env_vanilla/lib/python3.9/site-packages/easy-install.pth . My question is: who installed THAT? – Michael Tiemann Nov 28 '22 at 21:41
  • 1
    @MichaelTiemann if it was a Conda package, then the file should be described in the JSON file for the package found in the environment's `conda-meta/` folder. Something like `grep 'easy-install.pth' /Users/michael/miniconda3/envs/itr_env_vanilla/conda-meta/*` should identify the package. – merv Nov 28 '22 at 22:04
  • Option 3 is the easiest one and it has not given me any issues. – DaveR Mar 08 '23 at 10:37