Assembling Python module on fly, dynamic import

Question

I'm trying to get myself familiar with importlib hooks. I want to implement an ability to directly import non-pythonic files written in other language, and maintain source maps, so raising SyntaxErrors with line numbers will still give meaningful stacktraces.

My approach to loading foreign files is assembling Pythonic source, then compiling it and executing it in the desired context.

I've read in the documentation that implementing importlib.abc.SourceLoader seems to be my choice — however, the only method that is being called is exec_module. At that stage, to my understanding, module object is fully ready. So why are get_source, get_data, get_code and others are not called?

My stub implementation:

import sys
import os

import importlib.abc
import importlib.machinery

class MyFinder(importlib.abc.MetaPathFinder):
    def __init__(self):
        pass

    def find_spec(self, fullname, path, target=None):
        print('find_spec', fullname, path, target)

        # filename = '{}.npy'.format(fullname)
        # if not os.path.exists(filename):
        #     return

        if fullname != 'foobar':
            return

        filename = 'foobar://ponyworld/foo.py'

        spec = importlib.machinery.ModuleSpec(
            name = fullname,
            loader = MyLoader(fullname, path, target),
            origin = filename,
            loader_state = 1234,
            is_package = False,
        )

        return spec

class MyLoader(importlib.abc.SourceLoader):
    def __init__(self, fullname, path, target):
        pass

    def get_data(self, path):
        print('get_data', path)

    def get_filename(self, fullname):
        print('get_filename', fullname)

    def path_stats(self, path):
        print('path_stats', path)

    def set_data(self, path, data):
        print('set_data', path, data)

    def get_code(self, fullname):
        print('get_code', fullname)

    def exec_module(self, module):
        print('exec_module', module)
        print(dir(module))

    def get_source(self, fullname):
        print('get_source', fullname)

    def is_package(self, fullname):
        print('is_package', fullname)

sys.meta_path.append(MyFinder())

# import fake module to see if it works
import foobar

Since you don't do anything in the `Loader`'s `__init__` you can combine `MyLoader` and `MyFinder` into `MyImporter` by decorating `find_spec` as a `classmethod` The `__init__` then becomes the init for the `SourceLoader` parent. Supposedly this saves one from instantiating the `Loader`. Though this makes it a little less flexible if the `Finder` needs localized information, as this is only really passed in by instantiation. — Carel, Nov 24 '16 at 09:39

score 3 · Accepted Answer · answered Apr 12 '15 at 01:49

3

I misunderstood documentation and overlooked that importlib.abc.SourceLoader already provides implementations for get_code, exec_module, load_module, get_source and is_package. Those are to be overridden only in need, and only get_data and get_filename are the required minimum.

answered Apr 12 '15 at 01:49

toriningen

7,196
3
46
68

1

Don’t forget about `importlib.machinery.SourceFileLoader`! It is not an Abstract Base Class, and provides implementations for all of these. (Note the class name: Source*File*Loader. I once tried to replace `.abc.` with `.machinery.`, because at first glance it’s easy for your brain to assume it has the same class name as the `SourceLoader` from `importlib.abc`…but that’s not the case. ☻) – Zearin Apr 04 '16 at 16:24

Assembling Python module on fly, dynamic import

1 Answers1

Linked