import local package over global package

Question

I'm working on a support library for a large Python project which heavily uses relative imports by appending various project directories to sys.path.

Using The Hitchhiker's Guide to Packaging as a template I attempted to create a package structure which will allow me to do a local install, but can easily be changed to a global install later if desired.

One of the dependencies of my package is the pyasn1 package for the encoding and decoding of ASN.1 annotated objects. I have to include the pyasn1 library separately as the version supported by the CentOS 6.3 default repositories is one major version back and has known bugs that will break my custom package.

The top-level of the library structure is as follows:

MyLibrary/
  setup.py
  setup.cfg
  LICENSE.txt
  README.txt
  MyCustomPackage/
  pyasn1-0.1.6/

In my setup configuration file I define the install directory for my library to be a local directory called .lib. This is desirable as it allows me to do absolute imports by running the command import site; site.addsitedir("MyLibrary/.lib") in the project's main application without requiring our engineers to pass command line arguments to the setup script.

setup.cfg

[install]
install-lib=.lib

setup.py

setup(                                                                          
   name='MyLibrary',
   version='0.1a',
   package_dir = {'pyasn1': 'pyasn1-0.1.6/pyasn1'},   
   packages=[
             'MyCustomPackage',         
             'pyasn1',
             'pyasn1.codec',
             'pyasn1.compat','
              pyasn1.codec.ber',
             'pyasn1.codec.cer',
             'pyasn1.codec.der',
             'pyasn1.type'
            ],
   license='',
   long_description=open('README.txt').read(),
   data_files = []
)

The problem I've run into with doing the installation this way is that when my package tries to import pyasn1 it imports the global version and ignores the locally installed version.

As a possible workaround I have tried installing the pyasn1 package under a different name than the global package (eg pyasn1_0_1_6) by doing package_dir = {'pyasn1_0_1_6':'pyasn1-0.1.6/pyasn1'}. However, this fails since the imports used internally to the pyasn1 package do not use the pyasn1_0_1_6 name.

Is there some way to either a) force Python to import a locally installed package over a globally installed one or b) force a package to install under a different name?

Stefano Sanfilippo · Accepted Answer · 2013-05-06T18:55:50.397

Use virtualenv to ensure that your application runs in a fully known configuration which is independent from the OS version of libraries.

EDIT: a quick (unix) solution is setting the PYTHONPATH environment variable, which works just like PATH for Python modules (module loaded from first path in which is found, so simply append you directory at the beginning of the PYTHONPATH). Anwyay, I strongly recommend you to proceed with virtualenv, since it was specifically engineered for handling situations like the one you are facing.

Rationale

The process is easily automatable if you write a setuptools script specifying dependencies with install_requires. For a complete example, refer to this one I wrote

Setup

Note that you can easily insert the steps below in a setup.sh shell script.

First create a virtualenv and enter it:

$ virtualenv $name
$ cd $name

Activate it:

$ source bin/activate

Now cd to your project directory and run the installer script:

$ cd $my_project_dir
$ python ./setup.py --prefix $path_to_virtualenv

Note the --prefix $path_to_virtualenv, which is used to tell the script to install in the virtualenv instead of system-wide. Call this after activating the virtualenv. Note that all the depencies are automatically downloaded and installed in the virtualenv.

Then you are done. When you want to leave the virtualenv, issue:

$ deactivate

On subsequent calls, you will only need to activate the virtualenv (step 2), maybe using a runawesomeproject.sh if you really want.

As noted on the virtualenv website, you should use virtualenv >= 1.9, as the previous versions did not download dependencies via HTTPS. If you consider plain HTTP to be sufficient, then any version should do.

You might also try relocatable virtualenvs: setup it and copy the folder to your host. Anyway, note that this feature is still experimental.

Thanks for taking the time to respond - I've been stuck on this for a couple of days and appreciate the help. My concern with this solution is that it may place a burden on the users to properly setup and activate a virtual environment on their production machines to run the program. Is there a relatively simple way to script the install, creation and use (sourcing) of a virtual environment? — , May 05 '13 at 23:18
This doesn't solve my particular problem, but I think it's a reasonable answer given the scope of the question. — , Jun 08 '13 at 02:11

import local package over global package

1 Answers1

Rationale

Setup