Import local testing version of a Python namespace package

Question

I am wondering how I can import a local testing version of a Python 2.7 namespace package. In this example the package Ska.engarchive is a namespace package under the Ska root. (This structure is forced on me by legacy).

This example shows that even after setting sys.path to start with the local directory, the installed version of the package gets imported, not the local version.

Python 2.7.9 |Continuum Analytics, Inc.| (default, Apr 14 2015, 12:54:25)
... 
In [1]: import sys
In [2]: sys.path.insert(0, '.')
In [3]: import Ska.engarchive.fetch_eng as fetch
In [4]: fetch.__file__
Out[4]: '/proj/sot/ska/arch/x86_64-linux_CentOS-5/lib/python2.7/site-packages/Ska.engarchive-0.36.2-py2.7.egg/Ska/engarchive/fetch_eng.pyc'

I think the problem is related to the way namespace packages are implemented in Python 2, and somehow the namespace paths are always at the front of the list no matter what. But maybe there is some workaround? I spent some time digging around the site package docs, but maybe I just didn't see the right thing.

The above example is using the Anaconda Python distribution. Interestingly if I use a really old build of Python from ActiveState, the example has the desired outcome of importing the local package:

Python 2.7.1 (r271:86832, Feb  7 2011, 11:30:54) 

In [1]: import sys
In [2]: sys.path.insert(0, '.')
In [3]: import Ska.engarchive.fetch_eng as fetch
In [4]: fetch.__file__
Out[4]: './Ska/engarchive/fetch_eng.pyc'

Any help would be hugely appreciated!

Answer 1

I was able to replicate this behavior using setuptools's method for namespace packages (but not the standard pkgutil method). Setuptools will import the installed package while pkgutil will import the local package. Setuptools appears to load __path__ backwards from what you would expect (installed first, local second). E.g., (see example at bottom for definition of nstest)

>>> import nstest
>>> nstest.__path__
['/home/caleb/.local/lib/python2.7/site-packages/nstest.foo-0.0.0-py2.7.egg/nstest', 'nstest']

The way to get around this is to add your local package in the front of __path__ to the right-most namespace package. E.g.,

>>> import nstest
>>> nstest.__path__.insert(0, 'nstest')
>>> from nstest import foo
>>> foo.__file__
'nstest/foo/__init__.py'

Since you say in your question that Ska.engarchive is a namespace package, in your interpreter you want to do the following:

>>> import Ska.engarchive
>>> Ska.engarchive.__path__.insert(0, 'Ska/engarchive')
>>> import Ska.engarchive.fetch_eng as fetch
>>> fetch.__file__
'Ska/engarchive/fetch_eng.pyc' # This should be outputted

---

Namespace test using setuptools's pkg_resources

Directory structure:

nstest.foo/
├─ setup.py
└─ nstest/
   ├─ __init__.py
   └─ foo/
      └─ __init__.py

nstest.foo/setup.py:

from setuptools import setup, find_packages
setup(name='nstest.foo', packages=find_packages())

nstest.foo/nstest/__init__.py:

__import__('pkg_resources').declare_namespace(__name__)

nstest.foo/nstest/foo/__init__.py:

# empty

Install:

$ python2 setup.py build
$ python2 setup.py install --user

Test:

$ python2
>>> from nstest import foo
>>> foo.__file__
'/home/caleb/.local/lib/python2.7/site-packages/nstest.foo-0.0.0-py2.7.egg/nstest/foo/__init__.pyc'