What is happening in m5/objects/__init__.py file gem5

Question

I am new with gem5 simulator. I was reading the documentation (http://www.m5sim.org/Configuration_/_Simulation_Scripts) trying to understand how everything is implemented. When they write about Python classes they say the following:

gem5 provides a collection of Python object classes that correspond to its C++ simulation object classes. These Python classes are defined in a Python module called "m5.objects". The Python class definitions for these objects can be found in .py files in src, typically in the same directory as their C++ definitions.

To make the Python classes visible, the configuration file must first import the class definitions from the m5 module

In the m5/objects directory there is only one file "__init__.py". This is the code:

from __future__ import print_function
from __future__ import absolute_import

from m5.internal import params
from m5.SimObject import *

try:
    modules = __loader__.modules
except NameError:
    modules = { }

for module in modules.keys():
    if module.startswith('m5.objects.'):
        exec("from %s import *" % module)

Normally I don't program with Python so perhaps that is the problem, but I haven't fully understood what is going on here. In this other post Python's __loader__, what is it? they speak about what loader means but I feel I am missing something. Any help would be appreciated. Thanks in advance.

Answer 1

The __loader__

Consider the following code:

import sys
class FooImporter:
    def find_module(self, module_name, package_path):
        return self if module_name == 'foo' else None

    def load_module(self, module_name):
        print('FooImporter is working.')
        sys.modules[module_name] = __import__('sys')

# This activates the importer
sys.meta_path.append(FooImporter())
# This should trigger our importer to import 'foo'
import foo
# Show what we've just got
print(foo)

This will result in output:

FooImporter is working.
<module 'sys' (built-in)>

As long as you do not have a python module named foo in PYTHONPATH.

Python Import Hook (PEP 302) allows us to customize the behavior of import. In the above example, module foo was said to be found and handled by the FooImporter. Note the importer will create the module foo as an alias of sys. The complete importer (unlike the simplified one we've seen) will be responsible for setting the __loader__ attribute for the imported module to the importer itself.

Gem5's import hook

Back to your question, gem5 is using the same mechanism for loading SimObject's by its design of modulization. You can find the very importer class at src/python/importer.py with the class name CodeImporter.

When the module m5.object was being imported, say,

from m5.objects import Root

The CodeImporter will be responsible for handling the import task, in which the __loader__ attribute will be set for the imported module (in this case m5.objects). If you try printing __loader__ in m5/objects/__init__.py, you'll get something like:

<importer.CodeImporter object at 0x7f4f58941d60>

The __loader__.modules is a dictionary containing gem5 maintained SimObjects where each item will be added by addModule() calls from src/sim/init.cc.

As long as a SimObject's C++ correspondence has called the constructor for EmbeddedPython, it will be added to a list so the gem5 initialization will remember to add it to the instance of CodeImporter. For example, one should be able to find a Root.py.cc file in the build folder that registers the Root object. The loop at the end of m5/object/__init__.py is just importing a list of known SimObject's by this mechanism.

I think this should be sufficient to give someone a picture of the underlying magic and (hopefully) resolve their curiosity.