Load package but package with same name is already loaded

Question

I have two versions of the same Python package. I need from a module in a subpackage in the current version to be able to call a function inside the old version of the package (which copied itself in the past)

Where I am now:

now/
  package/
    __init__.py
    subpackage/
      __init__.py
      module.py -> "import package.subpackage.... <HERE>"
    subpackage2/
      ...
    ...

The old version:

past/
  package/
    __init__.py
    subpackage/
      __init__.py
      module.py -> "import package.subpackage; from . import module2; .... def f(x) ..."
      module2.py
    subpackage2/
      ...
    ...

I need to import in <HERE> the "old" f and run it.

Ideally

• the function f should live its life inside the old package without knowing anything about the new version of the package
• the module in the new package should call it, let it live its life, get the results and then forget altogether about the existence of the old package (so calling "import package.subpackage2" after letting f do her thing should run the "new" version)
• doing that should not be terribly complex

The underlying idea is to improve reproducibility by saving the code that I used for some task along with the output data, and then being able to run parts of it.

Sadly, I understood this is not a simple task with Python 3, so I am prepared to accept some sort of compromise. I am prepared to accept, for example that after running the old f(x) the name package in the "new" code will be bound to the old.

EDIT

I tried in two ways using importlib. The idea was to create an object mod and then doing f = getattr(mod, "f"), but it doesn't work

1. Changing sys.path to ['.../past/package/subpackage'] and then calling importlib.import_module('package.subpackage.module') . The problem is that it will load the one in "now" even with the changed sys.path, probably because the name package is already in sys.modules
2. spec = importlib.util.spec_from_file_location("module", "path..to..past..module.py")) module = importlib.util.module_from_spec(spec) spec.loader.exec_module(module) In that case relative imports (from . import module2.py) won't work, giving the error "attempted relative import with no known parent package"

Answer 1

There is one way this could work quite simply, but you will have to make a few modifications to your old package.

You can simply create a file in now/package/old/__init__.py containing:

__path__ = ['/absolute/path/to/old/package']

In the new package, you can then do:

from package.old.package.subpackage.module import f as old_f

The catch here is that the old package tries to import its own packages using absolute import, it is going to load stuff from the new packages instead. So the old package will have to only use relative imports when importing stuffs from its own package or you'll have to prepend package.old to all absolute imports that the old package was doing.

If you are fine with modifying the old packages in this way, then that should be fine. If that limitation would not work for you, then read on.

If you are really, really sure that for some reasons don't want to modify the old packages. Then let's do some black magic, you'd want to replace builtins.__import__ with your own version that returns different modules depending on who is doing the importing. You can figure out who is doing the importing by inspecting the call stack.

For example, this is how you might do it (tested on Python 3.6):

import builtins
import inspect
import package.old

old_package_path = package.old.__path__[0]

OUR_PACKAGE_NAME = 'package'
OUR_PACKAGE_NAME_WITH_DOT = OUR_PACKAGE_NAME + '.'


def import_module(name, globs=None, locs=None, fromlist=(), level=0):
    # only intercept imports for our own package from our old module
    if not name.startswith(OUR_PACKAGE_NAME_WITH_DOT) or \
            not inspect.stack()[1].filename.startswith(old_package_path):
        return real_import(name, globs, locs, fromlist, level)

    new_name = OUR_PACKAGE_NAME + '.old.' + name[len(OUR_PACKAGE_NAME_WITH_DOT):]
    mod = real_import(new_name, globs, locs, fromlist, level)
    return mod.old

# save the original __import__ since we'll need it to do the actual import
real_import = builtins.__import__
builtins.__import__ = import_module

builtins.__import__ gets called on any import statements encountered by the interpreter, and the call is not cached so you can return different things every time it is called even when they use the same name.

---

The following is my old answer, here for historical purpose only

I don't quite get what you're trying to do, but this is likely possible to do in Python 3 by using importlib.

You would just create a module loader that loads your module from an explicit filepath.

There's also an invalidate_caches() and reload() function which may be useful, though you may not need them.