Conditionally mapped module import

Question

I'm currently working on a abstract set of modules for Python 2.7, that I will ship as a python package:

 myabstractpkg
   - abstract
      - core
       - logging
       - ...
     - nodes
     - ...

The modules will then be implemented in a totally different set of packages:

 myimppkg
   - implementation
     - core
       - logging
       - ...
     - nodes
     - ...

At runtime however, I want to be able to do always imports like this in my tools that use the implemented modules:

from myabstractpkg.api import nodes
from myabstractpkg.api.core.logging import Logger

This way the developer always imports from the "virtual" api module which would then decide where to actually point the importer.

I know I might somehow be able to hack it together by modifying the modules dict:

from myimppkg import implementation
sys.modules["myabstractpkg.api"] = implementation

or doing a clever import for everything in __init__.py of myabstractpackage.api, but this feels a bit fragile to me.

I wonder if you guys have some input on what the best approach to do this is. I might be on a really ugly track with this whole remapping modules thing, so if you guys have any smarter, more pythonic solutions, for my API abstraction, implementation, usage approach, I would love to hear them.

Check out entry points ... https://amir.rachum.com/blog/2017/07/28/python-entry-points/ — Matthew Story, Jul 01 '18 at 20:22
Hi Matthew, I'm already using entry points to register my implementation packages. However, I don't know how to approach a mapping of those imports, so developers would always every use that one api package to import from. In ```Application A``` they should then automatically use ```Implementation A```, while in ```Application B``` the scripts should always use ```Implementation B```. — Tim Lehr, Jul 01 '18 at 20:28
cpython's [`decimal` module does this](https://github.com/python/cpython/blob/6f0eb93183519024cb360162bdd81b9faec97ba6/Lib/decimal.py). It's not for the same reason (it falls back on a python implementation if the c extension isn't found), but the `try/except ImportError` approach may be worth considering. — jedwards, Jul 01 '18 at 20:33
@Tim can you add your existing entry-points code to the question so we can get a sense of what's already done? — Matthew Story, Jul 01 '18 at 20:35
@PM2Ring I'm supporting environments that only support Python 2.7 (3D tools like Autodesk Maya and SideFX Houdini). — Tim Lehr, Jul 01 '18 at 20:40
Fair enough. But it's a worry that something as well-known as Maya doesn't support Python 3 yet. Don't they know that Python 2 reaches its End of Life in 2020? — PM 2Ring, Jul 01 '18 at 20:45
@MatthewStory I would first need to chop it to bits, as I built a small extension framework around it. It doesn't do anything necessarily related to the question though. Let's say for now that everything is in one package (abstraction, implementations and api module) and I know how to get the module name and / or path to the source files of the implementation. How would I dynamically map ```api``` between those implementations? — Tim Lehr, Jul 01 '18 at 20:46
@PM2Ring the industry is preparing to switch, but is cutting it really close. The major applications are expected to make the move in CY2020. https://www.vfxplatform.com — Tim Lehr, Jul 01 '18 at 20:48

Matthew Story · Accepted Answer · 2018-07-02T10:53:07.217

I believe that you are best served by making use of the entry_points capabilities of setuptools. So in the setup.py of one of your concrete implementations, you would define an entry_points like this:

setup(
    name="concrete_extension"
    entry_points={
        "abstract_pkg_extensions": [
            "concrete = concrete_extension"
        ]
    }
}

Then you could have an extension module in your abstract package which does something like this:

import pkg_resources
import os
import sys

from . import default

extensions = { "default": default }
extensions.update({e.name: e.load() for e in 
pkg_resources.iter_entry_points("my_pkg_extensions")})

current_implementation_name = None
current_implementation = None

def set_implementation(name):
    global current_implementation_name, current_implementation
    try:
        current_implementation = extensions[name]
        current_implementation_name = name

        # allow imports like from foo.current_implementation.bar import baz
        sys.modules["{}.current_implementation".format(__name__)] = current_implementation
        # here is where you would do any additional stuff
        # -- e.g. --
        # from . import logger
        # logger.Logger = current_implementation.Logger
    except KeyError:
        raise NotImplementedError("No implementation for: {}".format(name))

set_implementation(os.environ.get("CURRENT_IMPLEMENTATION_NAME", "default"))

You can then use ext.current_implementation to access the current implementation, and either set your implementation in the program's environment prior to import, or you can explicitly call set_implementation in your code before importing any sub-modules that make use of ext.current_implementation.

For more on the sys.modules entry and how it works in detail, see this question.

@Tim i'm making a lot of assumptions here, happy to edit to provide more clarity for you if necessary, but this is similar to patterns I have used in the past. — Matthew Story, Jul 01 '18 at 21:10
Thank you for the in-depth example! That's pretty close to what I was trying the other night. This approach works pretty well. However while this works ```import scdcc; scdcc.api.core```, I will still end up with an ImportError when I try to import a submodule in the namespace of another module. ```from scdcc.api import core # ImportError: No module named api```. Any idea on how to solve this? — Tim Lehr, Jul 02 '18 at 09:42
Because `api` is a variable you can't traverse it with `.` notation at import time like that afaik, so it's a limitation of this pattern ... though again, actual code would help here in determining what if anything can be done. — Matthew Story, Jul 02 '18 at 10:17
I just made it work by combining the entry point approach with the import approach explained here: https://stackoverflow.com/a/24324577/9288624 I needed to add a directory named ```api``` with an empty ```__init__.py``` and then I set the value for the appropriate dict entry: ```sys.modules["scdcc.api"] = implementation_module``` — Tim Lehr, Jul 02 '18 at 10:44

Conditionally mapped module import

1 Answers1