Where is Python path being manipulated?

Question

On one server I'm able to import a library, on a second one I'm not. The library I need to import is shared via NFS and reachable by both servers.

I found out that the Python path is different between the two servers, but the Python binary is the same, installed from standard ubuntu 16.04 repos, and $PYTHONPATH is unset on both servers.

Server 1:

$ echo $PYTHONPATH

$ python
>>> import sys; sys.path
['', '/usr/lib/python2.7', '...', '/usr/local/lib/python2.7/dist-packages', '/home/user/app/src/python', '/usr/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages/wx-3.0-gtk2']

Server 2:

$ echo $PYTHONPATH

$ python
>>> import sys; sys.path
['', '/usr/lib/python2.7', '...', '/usr/local/lib/python2.7/dist-packages', '/usr/lib/python2.7/dist-packages']

How do I understand where is this library defined, in order to replicate the same behaviour to server 2?

The curious fact is that the library is placed between the paths /usr/local/lib and /usr/lib.

Edit: The python binary is the same (same version 2.7.12, and the checksums match) and it's installed from the ubuntu repos in the standard location /usr/bin/python.
I've played with the library site and all variables are the same (site.ENABLE_USER_SITE returns True).
I know I can manipulate the paths inside the script, or specify PYTHONPATH in /etc/profile.d/ for all servers, but I'd like to know where is the difference.

This looks like a DIY deployment. You should consider creating [virtual environments](https://docs.python.org/3/tutorial/venv.html) for your application on both servers and install the library and all depenencies properly, with `pip` for example. — Klaus D., Feb 21 '20 at 12:07
It surely is a DIY deployment, but I cannot play freely with it. I know how to use virtualenvs, thanks. I'd like to know "where is the python path manipulated", since that dir is for sure not part of the default site installation. — Alberto Chiusole, Feb 21 '20 at 12:29

Łukasz Ślusarczyk · Answer 1 · 2020-02-21T12:45:48.050

0

Sys module documentation says that sys.path is

A list of strings that specifies the search path for modules. Initialized from the environment variable PYTHONPATH, plus an installation-dependent default

See: https://docs.python.org/3/library/sys.html

So there must be different installation-dependent default. Check exact python versions. Maybe python modules binaries come from different apt repositories. Maybe python commands come from different locations in both servers - check it by:

which python

Also check site module - it seems that it controls extra paths. See: http://docs.python.org/3/library/site.html

I would import it and check site.ENABLE_USER_SITE Maybe it will give you some clue on where the difference is

edited Feb 21 '20 at 12:45

answered Feb 21 '20 at 12:17

Łukasz Ślusarczyk

1,775
11
20

The python binary is exactly the same, I've checked with md5 and they match. It's probably something else other than the binary. – Alberto Chiusole Feb 21 '20 at 12:26
1

I would also check *site module* - it seems that it controls extra paths. See: https://docs.python.org/3/library/site.html I would import it and check site.ENABLE_USER_SITE Maybe it will give you some clue on where the difference is. – Łukasz Ślusarczyk Feb 21 '20 at 12:36
I've checked all the variables and methods inside `site`, and they are all the same on both servers, so I don't really know what else could it be. – Alberto Chiusole Feb 21 '20 at 14:14

score 0 · Accepted Answer · answered Mar 23 '20 at 15:42

I've discovered the existence of path configuration files in Python. From the doc of the site module on Python 3:

It starts by constructing up to four directories from a head and a tail part. For the head part, it uses sys.prefix and sys.exec_prefix; empty heads are skipped. For the tail part, it uses the empty string and then lib/site-packages (on Windows) or lib/pythonX.Y/site-packages (on Unix and Macintosh). For each of the distinct head-tail combinations, it sees if it refers to an existing directory, and if so, adds it to sys.path and also inspects the newly added path for configuration files.
...
A path configuration file is a file whose name has the form name.pth and exists in one of the four directories mentioned above; its contents are additional items (one per line) to be added to sys.path. Non-existing items are never added to sys.path, and no check is made that the item refers to a directory rather than a file. No item is added to sys.path more than once. Blank lines and lines beginning with # are skipped. Lines starting with import (followed by space or tab) are executed.

In my case there was a file in /usr/local/lib/python2.7/dist-packages/easy-install.pth which was missing on the second server.

Disabling the automatic import of those files with python -S may turn out to be useful while debugging.

Unfortunately I haven't found a way to retrieve a list of directories where the path configuration files are read from.

Where is Python path being manipulated?

2 Answers2