Python metaclass, execfile, inspect, globals(), namespace

Question

When the following script is run directly, it operates as anticipated:

import inspect

__module__ = "__main__"
__file__ = "classes.py"
test_str = "test"

class met(type):
    def __init__(cls, name, bases, dct):
        setattr(cls, "source", inspect.getsource(cls))
        #setattr(cls, "source", test_str)
        super(met, cls).__init__(name, bases, dct)

class ParentModel(object):
    __metaclass__ = met
    def __init__(self):
        super(object, self).__init__(ParentModel.__class__)
    def setsource(self):
        self.source = inspect.getsource(self.__class__)
        #self.source = test_str
    def getsource(self):
        return self.source

class ChildB(ParentModel):
    name = "childb"
    pass

class ChildA(ChildB):
    name = "childa"
    pass

class ChildC(ChildA):
    name = "childc"
    pass

The difficulty arises when attempting to run this script through exec or execfile in the python shell or another script. For instance:

>>> execfile("classes.py")

Runs without issue, however:

>>> ns = {}
>>> execfile("classes.py", ns)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "classes.py", line 13, in <module>
    class ParentModel(object):
  File "classes.py", line 9, in __init__
    setattr(cls, "source", inspect.getsource(cls))
  File "D:\Python27\lib\inspect.py", line 701, in getsource
    lines, lnum = getsourcelines(object)
  File "D:\Python27\lib\inspect.py", line 690, in getsourcelines
    lines, lnum = findsource(object)
  File "D:\Python27\lib\inspect.py", line 526, in findsource
    file = getfile(object)
  File "D:\Python27\lib\inspect.py", line 408, in getfile
    raise TypeError('{!r} is a built-in class'.format(object))
TypeError: <module '__builtin__' (built-in)> is a built-in class

This results in an error, which is confusing given dictionaries are accepted for the global namespace argument of execfile. But:

>>> execfile("classes.py", globals())

Again, runs without issue, though:

>>> ns = dict(globals())
>>> execfile("classes.py", ns)
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "classes.py", line 13, in <module>
    class ParentModel(object):
  File "classes.py", line 9, in __init__
    setattr(cls, "source", inspect.getsource(cls))
  File "D:\Python27\lib\inspect.py", line 701, in getsource
    lines, lnum = getsourcelines(object)
  File "D:\Python27\lib\inspect.py", line 690, in getsourcelines
    lines, lnum = findsource(object)
  File "D:\Python27\lib\inspect.py", line 526, in findsource
    file = getfile(object)
  File "D:\Python27\lib\inspect.py", line 408, in getfile
    raise TypeError('{!r} is a built-in class'.format(object))
TypeError: <module '__builtin__' (built-in)> is a built-in class

From the traceback, its related to inspect, however it should error on execfile("classes.py") or execfile("classes.py", globals()) as well.

So, in terms of this error, how is dict(globals()) != globals() and why does this cause this error ?

EDIT: Reader should refer to both Martijn Pieters and Lennart Regebro answers for a complete picture.

Answer 1

When you execute a python file with execfile() you are executing it in the current namespace. The REPL namespace is a built-in module:

>>> import sys
>>> sys.modules['__main__']
<module '__main__' (built-in)>

which means there is no sourcefile for inspect.getsource() to retrieve:

>>> sys.modules['__main__'].__file__
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
AttributeError: 'module' object has no attribute '__file__'
>>> import inspect
>>> inspect.getfile(sys.modules['__main__'])
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 403, in getfile
    raise TypeError('{!r} is a built-in module'.format(object))
TypeError: <module '__main__' (built-in)> is a built-in module

Your next problem is that because you use execfile the module set for your code is always going to be wrong. inspect.getsource() cannot determine where you defined the code, as execfile() bypasses the normal import mechanisms:

$ cat test.py
execfile('classes.py')
$ python test.py
Traceback (most recent call last):
  File "test.py", line 1, in <module>
    execfile('classes.py')
  File "classes.py", line 13, in <module>
    class ParentModel(object):
  File "classes.py", line 9, in __init__
    setattr(cls, "source", inspect.getsource(cls))
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 701, in getsource
    lines, lnum = getsourcelines(object)
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 690, in getsourcelines
    lines, lnum = findsource(object)
  File "/usr/local/Cellar/python/2.7.5/Frameworks/Python.framework/Versions/2.7/lib/python2.7/inspect.py", line 564, in findsource
    raise IOError('could not find class definition')
IOError: could not find class definition

At no point does your code work with execfile('classes.py', globals()) unless you run it directly from a file that has the same source code. On some platforms it may appear to work because execfile ends up triggering code that sets the __file__ attribute of the current module.

The only way you actually make this work is to

• Create a fake module object in sys.modules and give it a __file__ attribute that points to classes.py.
• Pass in a namespace to execfile() that sets __name__ to match the fake module object.

Demo:

>>> import sys
>>> import types
>>> sys.modules['fake_classes'] = types.ModuleType('fake_classes')
>>> sys.modules['fake_classes'].__file__='classes.py'
>>> ns = {'__name__': 'fake_classes'}
>>> execfile('classes.py', ns)
>>> >>> ns.keys()
['__module__', 'ChildA', '__builtins__', 'inspect', '__package__', 'met', 'ChildB', 'ChildC', 'ParentModel', '__name__', 'test_str']

Just to make it explicit, creating a copy of globals() only prevents execfile() from modifying your current module namespace (or your REPL namespace). There is otherwise no difference between the dictionaries being passed to execfile().

Answer 2

This issue isn't that dict(globals()) != globals(), because it is. The issue here is that the globals() in the context you execute execfile() is not the same globals() as in your classes.py.

When you pass in a namespace you replace the namespace that would otherwise be created. If you pass in a namespace, you'll get one created for you for the classes module. This means that __main__ will be the classes.py file. However, when you pass in the namespace of the file where execfile() is called, that namespace will be used instead, and __main__ will be that module.

That leads to inspect failing to find the class definition in the source code, because it's looking in the wrong file.

If you pass in an empty namespace, it will not find a __main__ at all, and the class will be assumed to be a builtin, with no source available, and an error to that effect will be raised.

In summary, the mistake you make here is to think of globals() and being global to the interpreter, when it's in fact global to the module.

From discussions with Marjtin, it's clear that things work slightly differently on OS X. That means you can not rely on this working even if you don't pass in a namespace.

And that then leads to the question of why you are doing this, and what you are actually trying to achieve.