Ironpython issue with import hook subpackage/submodule

Question

I have been struggling with something that I now think may be a bug in IronPython. If anyone could confirm this suspicion or set me straight I would be grateful.

The issue I am facing is when trying to apply import hooks using sys.meta_path or sys.path_hooks. IronPython seems to properly call my hooks when importing simple modules and packages. If I attempt to import a submodule or package such as foo.bar the hook is called to import foo but the hook is not called when attempting to import bar.

I have created the below code to replicate the issue. When run in CPython this same code does not suffer any issues.

import imp
import sys

class ReflectiveLoader2(object):

def __init__(self):
    print "Init RL2"

def find_module(self, fullname, path=None):
    print "find_module_RL2 called with fullname %s and path %s" % (fullname, path)
    self.path = path
    return self

def load_module(self, fullname):
    print "load_module_RL2 called with fullname %s" % fullname

    mod = imp.new_module(fullname)
    mod.__name__ = fullname
    mod.__loader__ = self
    #Simulate a couple scenarios for imports
    #Simulate a method
    if fullname == "foo":
        print "its foo"
        mod.__filepath__ = r"C:\not\a\real\path\foo.py"
        source = "def fooFunction(thing):print thing*3"

    #Simulate another method
    elif fullname == "bar":
        print "its bar"
        mod.__filepath__ = r"C:\not\a\real\path\bar.py"
        source = "def barFunction(thing):print thing*4"

    #Simulate package
    elif fullname == "baz":
        print "its baz"
        mod.__filepath__ = r"C:\not\a\real\path\baz\__init__.py"
        mod.__path__ = r"C:\not\a\real\path\baz"
        source = "print 'initializing baz stuff here'"

    #Simulate subpackage
    elif fullname == "baz.bat":
        print "its baz.bat"
        mod.__filepath__ = r"C:\not\a\real\path\baz\bat.py"
        mod.__path__ = r"C:\not\a\real\path\baz"
            source = "def Batfunc():print 'in baz.bat sub package'"

    #catchall
    else:
        print "Not foo, bar or baz its %s" % fullname
        source = ""

    exec source in mod.__dict__
    return mod


sys.meta_path = [ReflectiveLoader2()]

import foo
foo.fooFunction("ABC")

import bar
bar.barFunction("DEF")

import baz.bat
baz.bat.Batfunc()

Here is the output from the above code using both IronPython and CPython. I have run this code on two seperate systems just in case the install was bad on one of them

  C:\Users\Administrator\project>"C:\Program Files (x86)\IronPython 2.7\ipy.exe" ReflectiveLoader2.py
    Init RL2
    find_module_RL2 called with fullname foo and path None
    load_module_RL2 called with fullname foo
    its foo
    ABCABCABC
    find_module_RL2 called with fullname bar and path None
    load_module_RL2 called with fullname bar
    its bar
    DEFDEFDEFDEF
    find_module_RL2 called with fullname baz and path None
    load_module_RL2 called with fullname baz
    its baz
    initializing baz stuff here

    Traceback (most recent call last):
        File "ReflectiveLoader2.py", line 63, in <module>
        ImportError: No module named bat

 C:\Users\Administrator\project>C:\Python27\python.exe ReflectiveLoader2.py
    Init RL2
    find_module_RL2 called with fullname foo and path None
    load_module_RL2 called with fullname foo
    its foo
    ABCABCABC
    find_module_RL2 called with fullname bar and path None
    load_module_RL2 called with fullname bar
    its bar
    DEFDEFDEFDEF
    find_module_RL2 called with fullname baz and path None
    load_module_RL2 called with fullname baz
    its baz
    initializing baz stuff here
    find_module_RL2 called with fullname baz.bat and path C:\not\a\real\path\baz
    load_module_RL2 called with fullname baz.bat
    its baz.bat
    in baz.bat sub package

edit: I forgot to mention I tried to research if other people where having this issue and the closest thing I could find is this post

https://mail.python.org/pipermail/ironpython-users/2012-April/015879.html

From what I could tell nothing was ever done to resove the issue reported

macsec · Accepted Answer · 2015-05-22T12:18:48.463

After some additional troubleshooting I think this was mostly operator error. It apears CPython is slightly more forgiving in terms of a module's path attribute. If the above code is changed so that

mod.__path__ = r"C:\not\a\real\path\baz"

Looks like

mod.__path__ = [r"C:\not\a\real\path\baz",]

Everything works fine as is.

If someone is motivated enough to create similar type checking for IronPython I found the code that is not checking for a list type in the IronPython source file IronPython\Runtime\Importer.cs at around lines 151 and 120 as seen below

object path;
List listPath;
    if (scope.__dict__._storage.TryGetPath(out path) && (listPath = path as List) != null) {
        return ImportNestedModule(context, scope, new [] {name}, 0, listPath);
                }

If you set __path__ to a string (in lieu of the expected list) the conversion to a list fails and ImportNestedModule is never called.

Issue has been opened for the inconsistency between CPython and IronPython https://github.com/IronLanguages/main/issues/1202

Ironpython issue with import hook subpackage/submodule

1 Answers1