PYTHONPATH order on Ubuntu 14.04

Question

I have two computers running Ubuntu 14.04 server (let's call them A and B). B was initially a 10.04 but it has received two upgrades to 12.04 and 14.04. I do not understand why the python path is different on the two computers.

As you can see on the two paths below, the pip installation path /usr/local/lib/python2.7/dist-packages comes before the apt python packages path /usr/lib/python2.7/dist-packages on Ubuntu A, but it comes after on Ubuntu B.

This leads to several problems if a python package is installed both via apt and pip. As you can see below, if both python-six apt package and six pip package are installed, they may be two different library versions.

The installation of packages system is not always my choice, but might be some dependencies of other packages that are installed.

This problem could probably be solved with a virtualenv, but for reasons I will not detail, I cannot use virtualenv here, and must install pip packages system-wide.

Ubuntu A

>>> import sys, six
>>> sys.path
['',
 '/usr/local/bin',
 '/usr/lib/python2.7',
 '/usr/lib/python2.7/plat-x86_64-linux-gnu',
 '/usr/lib/python2.7/lib-tk',
 '/usr/lib/python2.7/lib-old',
 '/usr/lib/python2.7/lib-dynload',
 '/usr/local/lib/python2.7/dist-packages',
 '/usr/lib/python2.7/dist-packages',
 '/usr/lib/python2.7/dist-packages/PILcompat',
 '/usr/local/lib/python2.7/dist-packages/IPython/extensions']
>>> six
<module 'six' from '/usr/local/lib/python2.7/dist-packages/six.pyc'>

Ubuntu B

>>> import sys, six
>>> sys.path
['',
 '/usr/local/bin',
 '/usr/lib/python2.7/dist-packages',
 '/usr/lib/python2.7',
 '/usr/lib/python2.7/plat-x86_64-linux-gnu',
 '/usr/lib/python2.7/lib-tk',
 '/usr/lib/python2.7/lib-old',
 '/usr/lib/python2.7/lib-dynload',
 '/usr/local/lib/python2.7/dist-packages',
 '/usr/lib/python2.7/dist-packages/PILcompat',
 '/usr/local/lib/python2.7/dist-packages/IPython/extensions']
>>> six
>>> <module 'six' from '/usr/lib/python2.7/dist-packages/six.pyc'>

For both machines $PATH is the same, and $PYTHONPATH is empty.

• Why are those PYTHONPATHS different?

• How can I fix the pythonpath order in "Ubuntu B" so it will load pip packages before the system ones, in a system-wide way? Is there a apt package I should reinstall or reconfigure so the PYTHONPATH would prioritize pip packages ?

Answer 1

As we cannot explore into your system, I am trying to analysis your first question by illustrating how sys.path is initialized. Available references are where-does-sys-path-starts and pyco-reverse-engineering(python2.6).

The sys.path comes from the following variables(in order):

1. $PYTHONPATH (highest priority)
2. sys.prefix-ed stdlib
3. sys.exec_prefix-ed stdlib
4. site-packages
5. *.pth in site-packages (lowest priority)

Now let's describe each of these variables:

1. $PYTHONPATH, this is just a system environment variable.
2. & 3. sys.prefix and sys.exec_prefix are determined before any python script is executed. It is actually coded in the source Module/getpath.c.

The logic is like this:

IF $PYTHONHOME IS set:
    RETURN sys.prefix AND sys.exec_prefix as $PYTHONHOME
ELSE:
    current_dir = directory of python executable;
    DO:
        current_dir = parent(current_dir)
        IF FILE 'lib/pythonX.Y/os.py' EXSITS:
            sys.prefix = current_dir
        IF FILE 'lib/pythonX.Y/lib-dynload' EXSITS:
            sys.exec_prefix = current_dir
        IF current_dir IS '/':
            BREAK
    WHILE(TRUE)
    IF sys.prefix IS NOT SET:
       sys.prefix = BUILD_PREFIX
    IF sys.exec_prefix IS NOT SET:
       sys.exec_prefix = BUILD_PREFIX

4. & 5. site-packages and *.pth are added by import of site.py. In this module you will find the docs:

   This will append site-specific paths to the module search path. On Unix (including Mac OSX), it starts with sys.prefix and sys.exec_prefix (if different) and appends lib/python/site-packages as well as lib/site-python. ... ...

   For Debian and derivatives, this sys.path is augmented with directories for packages distributed within the distribution. Local addons go into /usr/local/lib/python/dist-packages, Debian addons install into /usr/{lib,share}/python/dist-packages. /usr/lib/python/site-packages is not used.

   A path configuration file is a file whose name has the form .pth; its contents are additional directories (one per line) to be added to sys.path. ... ...

And a code snippet for important function getsitepackages:

sitepackages.append(os.path.join(prefix, "local/lib",
                            "python" + sys.version[:3],
                            "dist-packages"))
sitepackages.append(os.path.join(prefix, "lib",
                            "python" + sys.version[:3],
                            "dist-packages"))

Now I try to fig out where may be this odd problem comes from:

1. $PYTHONPATH, impossible, because it is empty both A and B
2. sys.prefix and sys.exec_prefix, maybe, please check them and as well as $PYTHONHOME
3. site.py, maybe, check the file.

The sys.path output of B is quite odd, dist-package (site-package) goes before sys.exec_prefix (lib-dynload). Please try to investigate each step of sys.path initialization of machine B, you may find out something.

Very sorry that I cannot replicate your problem. By the way, about your question title, I think SYS.PATH is better than PYTHONPATH, which makes me misinterpretation as $PYTHONPATH at first glance.

Answer 2

If you peek into Python's site.py, which you can by opening /usr/lib/python2.7/site.py in a text editor.

The sys.path is augmented with directories for packages distributed within the distribution. Local addons go into /usr/local/lib/python/dist-packages, the global addons install into /usr/{lib,share}/python/dist-packages.

You can change the order by overriding this:

def getsitepackages():
    """Returns a list containing all global site-packages directories
    (and possibly site-python).

    For each directory present in the global ``PREFIXES``, this function
    will find its `site-packages` subdirectory depending on the system
    environment, and will return a list of full paths.
    """
    sitepackages = []
    seen = set()

    for prefix in PREFIXES:
        if not prefix or prefix in seen:
            continue
        seen.add(prefix)

        if sys.platform in ('os2emx', 'riscos'):
            sitepackages.append(os.path.join(prefix, "Lib", "site-packages"))
        elif os.sep == '/':
            sitepackages.append(os.path.join(prefix, "local/lib",
                                        "python" + sys.version[:3],
                                        "dist-packages"))
            sitepackages.append(os.path.join(prefix, "lib",
                                        "python" + sys.version[:3],
                                        "dist-packages"))
        else:
            sitepackages.append(prefix)
            sitepackages.append(os.path.join(prefix, "lib", "site-packages"))
        if sys.platform == "darwin":
            # for framework builds *only* we add the standard Apple
            # locations.
            from sysconfig import get_config_var
            framework = get_config_var("PYTHONFRAMEWORK")
            if framework:
                sitepackages.append(
                        os.path.join("/Library", framework,
                            sys.version[:3], "site-packages"))
    return sitepackages

Answer 3

One thing to mention here is that sometimes the *.pth files can totally mess up the priorities of where the sys.path is sourced from.

For instance, on my 14.04 ubuntu, /usr/local/lib/python2.7/dist-packages/easy-install.pth looks like this:

import sys; sys.__plen = len(sys.path)
/usr/lib/python2.7/dist-packages
import sys; new = sys.path[sys.__plen:]; del sys.path[sys.__plen:]; p = getattr(sys, '__egginsert', 0); sys.path[p:p] = new; sys.__egginsert = p + len(new)

Notice how /usr/lib/python2.7/dist-packages is prepended - even before PYTHONPATH - this means you can't really affect sys.path.

To get your /usr/local/lib/python2.7/dist-packages libraries to take precedence, one option is to edit /usr/local/lib/python2.7/dist-packages/easy-install.pth as follows:

import sys; sys.__plen = len(sys.path)
/usr/local/lib/python2.7/dist-packages
/usr/lib/python2.7/dist-packages
import sys; new = sys.path[sys.__plen:]; del sys.path[sys.__plen:]; p = getattr(sys, '__egginsert', 0); sys.path[p:p] = new; sys.__egginsert = p + len(new)

Answer 4

The simplest way is to use sys.path to make sure that you will have right order of paths added. sys.path gives out the list of paths that are available in the PYHTONPATH environment variable with the right order. If you want any path to have higher priority over others, just add it to the beginning of the list.

And you will also find this in the official docs:

A program is free to modify this list for its own purposes.

WARNING: Even though this gives better control over the priority, just make sure that whatever library you add does not mess with the system libraries. Else, your library will be searched first, as it is in the beginning of the list, and they may replace the system libraries. Just as an example, if you have written a library by the name os, after adding this to the sys.path, that library will be imported instead of Python's in-built. So take all the caution and also with a large amount of salt before jumping to this.

Answer 5

PYTHONPATH is environmental variable which you can set how ever you want,to add additional directories.You should not install Python packages manually,use pip.On older Ubuntu,probably you have manually installed modules before the upgrade.