2

I am working on a project where I need to be able to add new builders to buildbot dynamically. The build processes will all be the same (think microservices) but they'll work on different source code repositories and produce outputs that differ mainly based on the name of the source code repo.

It's pretty easy to get dynamic build steps ala travis (https://github.com/buildbot/buildbot_travis) or just dynamic build factories: http://docs.buildbot.net/latest/manual/cfg-buildfactories.html#dynamic-build-factories

I found a script in contrib that'll help create arbitrary changes by posting to the appropriate, enabled URL: https://github.com/buildbot/buildbot-contrib/blob/master/master/contrib/post_build_request.py

Which POSTS to this webhook: https://github.com/buildbot/buildbot/blob/master/master/buildbot/www/hooks/base.py

Which has to be enabled like so in master.cfg

c['www'] = dict(port=8010,
            plugins=dict(waterfall_view={}, console_view={}),
            change_hook_dialects={
                'base': True,
            })

Great, I got that far. And when I did so, I look in twistd.log and see this:

2017-05-22 20:43:29-0500 [_GenericHTTPChannelProtocol,17,127.0.0.1] Attempting to load module buildbot.www.hooks.base
2017-05-22 20:43:29-0500 [_GenericHTTPChannelProtocol,17,127.0.0.1] Got the following changes [{'category': 'release', 'codebase': None, 'repository': 'mikes_microservice', 'author': 'sandford', 'when': None, 'comments': u'post_build_request submission', 'project': 'mike_basic_stats', 'branch': None, 'revlink': None, 'properties': {}, 'files': [], 'revision': None}]
2017-05-22 20:43:29-0500 [_GenericHTTPChannelProtocol,17,127.0.0.1] Payload: {'category': ['release'], 'project': ['mikes_microservice'], 'repository': ['mikes_microservice'], 'comments': ['post_build_request submission'], 'author': ['sandford']}
2017-05-22 20:43:29-0500 [_GenericHTTPChannelProtocol,17,127.0.0.1] WARNING: change source is using deprecated self.master.addChange method; this method will disappear in Buildbot-1.0.0
2017-05-22 20:43:29-0500 [-] added change with revision None to database
2017-05-22 20:43:29-0500 [-] injected change Change(revision=None, who=u'sandford', branch=None, comments=u'post_build_request submission', when=1495503809, category=u'release', project=u'mikes_microservice', repository=u'mikes_microservice', codebase=u'')

Good, everything is coming together! Alas, nothing is triggered to actually build. My suspicion is that it is because I don't have that builder in my builders config:

c['builders'] = []
c['builders'].append(
    util.BuilderConfig(name="example",
      workernames=["example-worker"],
      factory=factory))

So that's the trick. I recognize that I could create this list dynamically at start-up but we're making new microservices all the time and I'm a touch nervous about having to constantly restart.

Just to see how far I could get I added a BuilderConfig (and a scheduler entry) and attempted again:

c['builders'].append(
    util.BuilderConfig(name="mikes_microservice",
      workernames=["example-worker"],
      factory=factory))

And the scheduler:

c['schedulers'] = []
c['schedulers'].append(schedulers.SingleBranchScheduler(
                            name="all",
                            change_filter=util.ChangeFilter(branch='master'),
                            treeStableTimer=None,
                            builderNames=["example",]))

c['schedulers'].append(schedulers.ForceScheduler(
                            name="force",
                            builderNames=["example", "mikes_microservice", ]))

Then I tried running the script again:

python post_build_request.py -P mikes_microservice  -r mikes_microservice --branch master

When I did that I got a build on the "example" project, not the "mikes_microservice" project which makes me think I might not understand the way that schedulers or changefilters are done, as I was definitely expecting the "mikes_microservice" project/builder to be triggered.

I then added "mikes_microservice" to the SingleBranchScheduler builderNames and I was able to trigger a build of "mikes_microservice"... and a build of "example" also.

Figuring that having multiple builders on the same scheduler was causing the problem, I split it out to a separate scheduler like so, to the same effect.

c['schedulers'].append(schedulers.SingleBranchScheduler(
                            name="mikes_microservice",
                            change_filter=util.ChangeFilter(branch='master'),
                            treeStableTimer=None,
                            builderNames=["mikes_microservice"]))

Clearly I'm missing something about how to get different projects or repositories to build differently -- at least via the webhooks -- and once I solve that problem I'm also interested in injecting new [whatever all it takes to make this happen] into buildbot at runtime in order to cause it to pick up new projects or repositories and build them without a restart.

Mike Sandford
  • 1,315
  • 10
  • 22

1 Answers1

3

Hi thanks for this excellent question. This is where the virtual_builders come into the play.

The idea is to create a single builder where all the buildrequests will attach to. You can call it bootstrap. The job of this bootstrap is to read a buildbot.yml, which will then configure the rest of the build.

eve is doing similar things https://bitbucket.org/scality/eve/src/

Then the idea is to have a virtual_builder_name set automatically to the name of your project. I think the best would be to have a special AnyBranchScheduler that automatically sets the virtual_builder_name. This is not something that is builtin in Buildbot. Please feel free to work on that.

tardyp
  • 1,142
  • 11
  • 9