In the context of a buildbot master.cfg file, how do I instantiate a class based on a string using native modules

Question

In stock python, I can do the following to instantiate a class based on a string containing its name:

#!/usr/bin/env python2.7
import sys
class Foo(object): pass
cls = getattr(sys.modules[__name__], 'Foo')
instance = cls()
print repr(instance)

Which outputs the following:

ahammond@af6119›~⁑ ./test.py
<__main__.Foo object at 0x1095b0a10>

I'd like to do something similar inside a buildbot master.cfg file, however the following (simplified)

class BaseBuild(object): pass

class BuildStashToSrpmsToRpmsToDepot(BaseBuild):

    def init(name, build_type):
        self.name = name

    def setup():
        pass  # actually does stuff here, but...

for build_name, build_options in config['builds'].iteritems():
    cls = getattr(sys.modules[__name__], build_options['build_type'])
    build = cls(name=build_name, **build_options)
    build.setup()

Produces

2015-03-11 18:39:24-0700 [-] error while parsing config file:
    Traceback (most recent call last):
      File "/opt/buildbot_virtualenv/lib/python2.7/site-packages/twisted/internet/defer.py", line 577, in _runCallbacks
        current.result = callback(current.result, *args, **kw)
      File "/opt/buildbot_virtualenv/lib/python2.7/site-    packages/twisted/internet/defer.py", line 1155, in gotResult
        _inlineCallbacks(r, g, deferred)
      File "/opt/buildbot_virtualenv/lib/python2.7/site-packages/twisted/internet/defer.py", line 1099, in _inlineCallbacks
        result = g.send(result)
      File "/opt/buildbot_git/master/buildbot/master.py", line 189, in startService
        self.configFileName)
    --- <exception caught here> ---
      File "/opt/buildbot_git/master/buildbot/config.py", line 156, in loadConfig
        exec f in localDict
      File "/home/buildbot/master.cfg", line 208, in <module>
        cls = getattr(sys.modules[__name__], build_options['build_type'])
    exceptions.AttributeError: 'module' object has no attribute 'BuildStashToSrpmsToRpmsToDepot'

2015-03-11 18:39:24-0700 [-] Configuration Errors:
2015-03-11 18:39:24-0700 [-]   error while parsing config file: 'module' object has no attribute 'BuildStashToSrpmsToRpmsToDepot' (traceback in logfile)

Phrased another way, I guess what I'm really asking is what is the temporary module used while loading a new master.cfg and how can I reference it?

I'm currently using a dictionary mapping of { 'class name': class_object } but I'd prefer something a little more native.

score 1 · Answer 1 · answered Mar 12 '15 at 16:45

Your problem here is: When running buildbot, your dunder name (__ name __ without the spaces...) when buildbot exec your config is buildbot.config, this is why 'module object has no attribute...'.

I think you could do what you want making the values of your build_options dictionary to be the class itself, not a string with the class name. Like this:

class BaseBuild(object): pass

class BuildStashToSrpmsToRpmsToDepot(BaseBuild):

    def init(name, build_type):
        self.name = name

    def setup():
        pass  # actually does stuff here, but...

# here the dict goes with key/class not key/class name
build_options = {'build_type': BuildStashToSrpmsToRpmsToDepot}

for build_name, build_options in config['builds'].iteritems():
    cls = build_options['build_type']
    build = cls(name=build_name, **build_options)
    build.setup()

Just in case, this is how buildbot exec master.cfg (module buildbot.config):

# ...
# execute the config file
localDict = {
    'basedir': os.path.expanduser(basedir),
    '__file__': os.path.abspath(filename),
}

# from here on out we can batch errors together for the user's
# convenience
global _errors
_errors = errors = ConfigErrors()

old_sys_path = sys.path[:]
sys.path.append(basedir)
try:
    try:
        exec f in localDict
    except ConfigErrors, e:
        for err in e.errors:
            error(err)
        raise errors
    except:
        log.err(failure.Failure(), 'error while parsing config file:')
        error("error while parsing config file: %s (traceback in logfile)" % (sys.exc_info()[1],),)
        raise errors
finally:
    f.close()
    sys.path[:] = old_sys_path
        _errors = None
# ...

score 1 · Accepted Answer · answered Mar 13 '15 at 17:17

Ok, so the problem is here:

cls = getattr(sys.modules[__name__], build_options['build_type'])

This does not work because exec makes it so that __name__ has the value "__builtin__". However, you can use globals() to get the current globals:

cls = globals()[build_options['build_type']]

For instance, if I add the following code into a brand new master.cfg file (the one automatically created by buildbot create-master master, renamed from master.cfg.sample):

# Load the configuration from somewhere.
import json
config = json.load(open("./config.json"))

class BaseBuild(object):
    pass

class Something(BaseBuild):

    def __init__(self, name, build_type):
        self.name = name

    def setup(self):
        print self.name, "something setup called"

class SomethingElse(BaseBuild):

    def __init__(self, name, build_type):
        self.name = name

    def setup(self):
        print self.name, "something else setup called"

for build_name, build_options in config['builds'].iteritems():
    cls = globals()[build_options['build_type']]
    build = cls(name=build_name, **build_options)
    build.setup()

And I create the following config.json file in the same directory as master.cfg:

{
    "builds": {
        "one": {
            "build_type": "Something"
        },
        "two": {
            "build_type": "SomethingElse"
        }
    }
}

Then when I run buildbot start master, I'll get these lines in the log:

2015-03-13 12:11:05-0400 [-] two something else setup called
2015-03-13 12:11:05-0400 [-] one something setup called

In the context of a buildbot master.cfg file, how do I instantiate a class based on a string using native modules

2 Answers2