Reloading packages (and their submodules) recursively in Python

Question

In Python you can reload a module as follows...

import foobar

import importlib
importlib.reload(foobar)

This works for .py files, but for Python packages it will only reload the package and not any of the nested sub-modules.

With a package:

• foobar/__init__.py
• foobar/spam.py
• foobar/eggs.py

Python Script:

import foobar

# assume `spam/__init__.py` is importing `.spam`
# so we dont need an explicit import.
print(foobar.spam)  # ok

import importlib
importlib.reload(foobar)
# foobar.spam WONT be reloaded.

Not to suggest this is a bug, but there are times its useful to reload a package and all its submodules. (If you want to edit a module while a script runs for example).

What are some good ways to recursively reload a package in Python?

Notes:

• For the purpose of this question assume the latest Python3.x

  (currently using importlib)

• Allowing that this may requre some edits to the modules themselves.
• Assume that wildcard imports aren't used (from foobar import *), since they may complicate reload logic.

Answer 1

Heres a function that recursively loads a package. Double checked that the reloaded modules are updated in the modules where they are used, and that issues with infinite recursion are checked for.

One restruction is it needs to run on a package (which only makes sense for packages anyway)

import os
import types
import importlib


def reload_package(package):
    assert(hasattr(package, "__package__"))
    fn = package.__file__
    fn_dir = os.path.dirname(fn) + os.sep
    module_visit = {fn}
    del fn

    def reload_recursive_ex(module):
        importlib.reload(module)

        for module_child in vars(module).values():
            if isinstance(module_child, types.ModuleType):
                fn_child = getattr(module_child, "__file__", None)
                if (fn_child is not None) and fn_child.startswith(fn_dir):
                    if fn_child not in module_visit:
                        # print("reloading:", fn_child, "from", module)
                        module_visit.add(fn_child)
                        reload_recursive_ex(module_child)

    return reload_recursive_ex(package)

# example use
import os
reload_package(os)

Answer 2

I've updated the answer from @ideasman42 to always reload modules from the bottom of the dependency tree first. Note that it will raise an error if the dependency graph is not a tree (i.e. contains cycles) as I don't think it will be possible to cleanly reload all modules in that case.

import importlib
import os
import types
import pathlib

def get_package_dependencies(package):
    assert(hasattr(package, "__package__"))
    fn = package.__file__
    fn_dir = os.path.dirname(fn) + os.sep
    node_set = {fn}  # set of module filenames
    node_depth_dict = {fn:0} # tracks the greatest depth that we've seen for each node
    node_pkg_dict = {fn:package} # mapping of module filenames to module objects
    link_set = set() # tuple of (parent module filename, child module filename)
    del fn

    def dependency_traversal_recursive(module, depth):
        for module_child in vars(module).values():

            # skip anything that isn't a module
            if not isinstance(module_child, types.ModuleType):
                continue

            fn_child = getattr(module_child, "__file__", None)

            # skip anything without a filename or outside the package
            if (fn_child is None) or (not fn_child.startswith(fn_dir)):
                continue

            # have we seen this module before? if not, add it to the database
            if not fn_child in node_set:
                node_set.add(fn_child)
                node_depth_dict[fn_child] = depth
                node_pkg_dict[fn_child] = module_child
                
            # set the depth to be the deepest depth we've encountered the node
            node_depth_dict[fn_child] = max(depth, node_depth_dict[fn_child])

            # have we visited this child module from this parent module before?
            if not ((module.__file__, fn_child) in link_set):
                link_set.add((module.__file__, fn_child))
                dependency_traversal_recursive(module_child, depth+1)
            else:
                raise ValueError("Cycle detected in dependency graph!")

    dependency_traversal_recursive(package, 1)
    return (node_pkg_dict, node_depth_dict)
    
# example use
import collections
node_pkg_dict, node_depth_dict = get_package_dependencies(collections)
for (d,v) in sorted([(d,v) for v,d in node_depth_dict.items()], reverse=True):
    print("Reloading %s" % pathlib.Path(v).name)
    importlib.reload(node_pkg_dict[v])

Answer 3

I'll offer another answer for the case in which you want to reload only a specific nested module. I found this to be useful for situations where I found myself editing a single subnested module, and reloading all sub-nested modules via a solution like ideasman42's approach or deepreload would produce undesired behavior.

assuming you want to reload a module into the workspace below

my_workspace.ipynb

import importlib
import my_module
import my_other_module_that_I_dont_want_to_reload

print(my_module.test()) #old result
importlib.reload(my_module)
print(my_module.test()) #new result

but my_module.py looks like this:

import my_nested_submodule

def test():
   my_nested_submodule.do_something()

and you just made an edit in my_nested_submodule.py:

def do_something():
   print('look at this cool new functionality!')

You can manually force my_nested_submodule, and only my_nested_submodule to be reloaded by adjusting my_module.py so it looks like the following:

import my_nested_submodule
import importlib
importlib.reload(my_nested_submodule)

def test():
   my_nested_submodule.do_something()