Fake Namespace Packages in Python 2

Question

PEP 302 -- New Import Hooks specifies ways to hook the import mechanism of Python. One of them is to create a module finder/loader and add it to sys.meta_path.

I seek to create a module finder that is capable of re-routing the import of sub-packages. So let's say if I write import mypackage.sub it should actually import the module mypackage_sub.

mypackage.sub                => mypackage_sub
mypackage.sub.another        => mypackage_sub.another
mypackage.sub.another.yikes  => mypackage_sub.another.yikes

My attempts to implement such a module finder where not successful. For now, I install the finder class in the main Python file, but my aim is to install it in the root package mypackage.

The finder will receive a call to .find_module(fullname, path) with 'mypackage', but never with 'mypackage.sub'. Therefore, I just get the following error for the code below.

Traceback (most recent call last):  
  File "test.py", line 63, in <module>  
    import mypackage.sub  
ImportError: No module named sub

import sys
sys.meta_path.append(_ExtensionFinder('mypackage'))
import mypackage.sub

Here's the code for the _ExtensionFinder class. I appreciate any solution that would allow me to put this _ExtensionFinder class into the root module mypackage.

class _ExtensionFinder(object):
    """
    This class implements finding extension modules being imported by
    a prefix. The dots in the prefix are converted to underscores and
    will then be imported.

    .. seealso:: PEP 302 -- New Import Hooks
    """

    def __init__(self, prefix):
        super(_ExtensionFinder, self).__init__()
        self.prefix = prefix

    def _transform_name(self, fullname):
        if self.prefix == fullname:
            return fullname
        elif fullname.startswith(self.prefix + '.'):
            newpart = fullname[len(self.prefix) + 1:]
            newname = self.prefix.replace('.', '_') + '_' + newpart
            return newname
        return None

    def find_module(self, fullname, path=None):
        print "> find_module({0!r}, {1!r})".format(fullname, path)

        newname = self._transform_name(fullname)
        if newname:
            return self

    def load_module(self, fullname):
        print "> load_module({0!r})".format(fullname)

        # If there is an existing module object named 'fullname'
        # in sys.modules, the loader must use that existing module.
        if fullname in sys.modules:
            return sys.modules[fullname]

        newname = self._transform_name(fullname)
        if newname in sys.modules:
            sys.modules[fullname] = sys.modules[newname]
            return sys.modules[newname]

        # Find and load the module.
        data = imp.find_module(newname)
        mod = imp.load_module(newname, *data)

        # The __loader__ attribute must be set to the loader object.
        mod.__loader__ = self

        # The returned module must have a __name__, __file__
        # and __package__ attribute.
        assert all(hasattr(mod, n) for n in ['__name__', '__file__', '__package__'])

        return mod

Edit: I found out that the right words to describe what I want to achieve is "namespace package"

You could just do in `mypackage.py` (or `__init__.py`): `import mypackage_sub as sub`, if that's all you need. — matsjoyce, Oct 02 '14 at 16:55
@matsjoyce It should be possible to add sub-packages dynamically, without having to touch `mypackage.py` at all :) — Niklas R, Oct 02 '14 at 16:59
I just found out about `pkgutil.extend_path()` which looks pretty useful already, unfortunately that would require to keep every separate namespace package in a different directory in `sys.path`, which should also not be a requirement. — Niklas R, Oct 02 '14 at 17:00
Where do you want the insertion / meta path modding to occur? In `__main__.py`? Or `mypackage_sub`? — matsjoyce, Oct 02 '14 at 17:04
@matsjoyce `mypackage` should mod the meta path so any sub-packages can be directly imported with no further hassling (since `mypackage` will be imported before it attempts to import `mypackage.sub` anyway) — Niklas R, Oct 02 '14 at 17:09

Niklas R · Answer 1 · 2014-10-02T19:51:27.520

Apparently, the __path__ attribute must be set, otherwise the import mechanism will not invoke the finders/loaders in sys.meta_path. So when the module was loaded, just do

        # __path__ must be set on the module, otherwise submodules
        # will not be loaded using the _ExtensionFinder.
        if not hasattr(mod, '__path__'):
            mod.__path__ = []

This also counts for the root module mypackage. Here is the full mypackage.py file.

__all__ = []

# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
# Extension import hook
# ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

import sys
import imp
import importlib

# The __path__ variable must be set in the module, otherwise the
# _ExtensionFinder will not be invoked for sub-modules.
__path__ = []

class _ExtensionFinder(object):
    """
    This class implements finding extension modules being imported by
    a prefix. The dots in the prefix are converted to underscores and
    will then be imported.

    .. seealso:: PEP 302 -- New Import Hooks
    """

    def __init__(self, prefix):
        super(_ExtensionFinder, self).__init__()
        self.prefix = prefix

    def _check_name(self, fullname):
        if self.prefix == fullname:
            return 1
        elif fullname.startswith(self.prefix + '.'):
            return 2
        return 0

    def _transform_name(self, fullname):
        value = self._check_name(fullname)
        if value == 1:
            return fullname
        elif value == 2:
            newpart = fullname[len(self.prefix) + 1:]
            newname = self.prefix.replace('.', '_') + '_' + newpart
            return newname
        return None

    def find_module(self, fullname, path=None):
        if self._check_name(fullname):
            return self

    def load_module(self, fullname):
        # If there is an existing module object named 'fullname'
        # in sys.modules, the loader must use that existing module.
        if fullname in sys.modules:
            return sys.modules[fullname]

        # Get the name of the module that actually needs to be
        # imported for this extension.
        newname = self._transform_name(fullname)

        # If it already exists, just fill up the missing entry for
        # the extension name.
        if newname in sys.modules:
            sys.modules[fullname] = sys.modules[newname]
            return sys.modules[newname]

        # Load the module and add additional attributes.
        mod = importlib.import_module(newname)
        mod.__loader__ = self

        # __path__ must be set on the module, otherwise submodules
        # will not be loaded using the _ExtensionFinder.
        if not hasattr(mod, '__path__'):
            mod.__path__ = []

        # The returned module must have a __name__, __file__
        # and __package__ attribute.
        assert all(hasattr(mod, n) for n in ['__name__', '__file__', '__package__'])

        return mod

_ext_finder = _ExtensionFinder('mypackage')
sys.meta_path.append(_ext_finder)

Fake Namespace Packages in Python 2

1 Answers1