(Buildbot) Can't activate virtualenv using ShellCommand

Question

Returns error "OSError : no Such file or directory". We were trying to activate our newly created virtual env venvCI using the steps in builder with shellCommand.Seems like we cant activate the virtualenv venvCI. Were only new in this environment so please help us.Thanks.

from buildbot.steps.shell import ShellCommand

factory = util.BuildFactory()

# STEPS for example-slave:

factory.addStep(ShellCommand(command=['virtualenv', 'venvCI']))

factory.addStep(ShellCommand(command=['source', 'venvCI/bin/activate']))

factory.addStep(ShellCommand(command=['pip', 'install', '-r','development.pip']))

factory.addStep(ShellCommand(command=['pyflakes', 'calculator.py']))

factory.addStep(ShellCommand(command=['python', 'test.py']))

c['builders'] = []
c['builders'].append(
    util.BuilderConfig(name="runtests",
      slavenames=["example-slave"],
      factory=factory))

@varesa I will try that one and notify you for the result. Thanks! — Ronet de Leon, Feb 08 '16 at 00:28

score 3 · Answer 1 · answered Feb 05 '16 at 11:47

Since the buildsystem creates a new Shell for every ShellCommand you can't source env/bin/activate since that only modifies the active shell's environment. When the Shell(Command) exits, the environment is gone.

Things you can do:

Give the environment manually for every ShellCommand (read what activate does) env={...}
Create a bash script that runs all your commands in a single shell (what I've done in other systems)

e.g.

myscript.sh:

#!/bin/bash

source env/bin/activate
pip install x
python y.py

Buildbot:

factory.addStep(ShellCommand(command=['bash', 'myscript.sh']))

blog post about the issue

score 2 · Answer 2 · answered Jun 20 '16 at 02:20

Another option is to call the python executable inside your virtual environment directly, since many Python tools that provide command-line commands are often executable as modules:

from buildbot.steps.shell import ShellCommand

factory = util.BuildFactory()

# STEPS for example-slave:

factory.addStep(ShellCommand(command=['virtualenv', 'venvCI']))

factory.addStep(ShellCommand(
    command=['./venvCI/bin/python', '-m', 'pip', 'install', '-r', 'development.pip']))

factory.addStep(ShellCommand(
    command=['./venvCI/bin/python', '-m', 'pyflakes', 'calculator.py']))

factory.addStep(ShellCommand(command=['python', 'test.py']))

However, this does get tiresome after a while. You can use string.Template to make helpers:

import shlex
from string import Template

def addstep(cmdline, **kwargs):
    tmpl = Template(cmdline)
    factory.addStep(ShellCommand(
        command=shlex.split(tmpl.safe_substitute(**kwargs))
    ))

Then you can do things like this:

addstep('$python -m pip install pytest', python='./venvCI/bin/python')

These are some ideas to get started. Note that the neat thing about shlex is that it will respect spaces inside quoted strings when doing the split.

(Buildbot) Can't activate virtualenv using ShellCommand

2 Answers2