deploying Python code with Puppet

Question

I'm trying to implement a deployment system for my web application, based on Puppet.

The application consists of several services (a couple of web servers: Django- and Tornado-based and a worker based on Celery). They live in different git repositories, hosted on github.

I've used Chef for my previous projects. In case of Chef, deploying such thing would be easy: I'd just use git resource to checkout necessary code for this particular node, and set everything else up.

However, I don't understand how it's done in Puppet. I haven't found a git type in documentation. I've tried Googling it, but from what I found it seems that checking out and updating git code from Puppet is not a usual task.

Clearly, my assumed workflow is somewhat unusual for Puppet. I could try something else. What's the natural way to deploy such apps with Puppet?

Possible duplicate of http://serverfault.com/questions/296299/how-to-deploy-applications-in-tar-gz-with-puppet — François Beausoleil, Apr 19 '13 at 20:31

score 3 · Answer 1 · answered May 25 '12 at 13:28

It's true that there is not native git resource type in Puppet, but I wouldn't call it an "unusual task".

There are a variety of solutions out there for implementing this yourself using define. We're using a git class that defines a git::repository resource type like this:

class git {
  define repository (
    $url,
    $workdir=undef,
    $branch=undef,
    $recursive=undef,
    $pull=true
  ) {
    if $workdir {
      $r_workdir = $workdir
    } else {
      $r_workdir = $name
    }

    if $branch {
      $branch_arg = "--branch $branch"
    }

    if $recursive {
      $recursive_arg = '--recursive'
    }

    exec { "clone-gitrepo-$name":
      creates => "$r_workdir",
      command => "/usr/bin/git clone $branch_arg $recursive_arg $url $r_workdir",
    }

    # This is here so other things can depend on
    # File[/path/to/working/directory].
    file { $r_workdir:
      ensure  => directory,
      require => Exec["clone-gitrepo-$name"],
    }

    if $pull {
      exec { "update-gitrepo-$name":
        require => Exec["clone-gitrepo-$name"],
        cwd     => "$r_workdir",
        command => '/usr/bin/git pull',
      }
    }
  }
}

We use it, for example, like this:

class openstack::controller::novnc {
  include git

  git::repository { '/opt/noVNC':
    url => 'git://github.com/cloudbuilders/noVNC.git',
  }
}

It performs a git pull operation every time Puppet runs, unless you set pull to false.

Note that I am not making any claims as to the correctness or robustness of this solution; it's what we're using and it works for us but there may be more feature-ful solutions out there.

You can find our code online here.

score 2 · Answer 2 · answered May 25 '12 at 13:31

The quick-and-dirty way is to use exec:

exec { 'install abcd':
  path    => ['/bin', '/usr/bin'],
  cwd     => '/usr/local',
  command => 'git clone http:/.../abcd',
  creates => '/usr/local/abcd',
}

It's also not difficult to create your own git resource, based on that:

define git ($package, $source) {
  exec {...}
}

However, I think that the clean way is to package all software you are using into packages (deb or yum or whatever your operating system uses), and upload it to the Puppet master. Then, when a server is being installed, it will find everything it needs on the Puppet master, rather than needing to connect to various git repositories (and pip repositories, and mercurial repositories, since if you start doing this trick for git, you'll do it for pip also). Then, installing will be way more reliable (one of the git/pip/hg connections could be temporarily down, or it might have moved), and will use less bandwidth, especially if you are installing your software on many servers. If you don't have time to spare to create packages, creating .tar.gz packages is an intermediate solution. Of course this also takes time, I know; I also use the quick-and-dirty way when I'm in a hurry.

score 1 · Accepted Answer · answered May 25 '12 at 17:59

1

I'd use fabric to deploy Python code -- in fact, I do. But if all you want is the git stuff, here's an example from my puppet configurations using the vcsrepo module:

vcsrepo { '/var/www/gitorious':
    ensure   => present,
    owner    => 'git',
    source   => 'git://gitorious.org/gitorious/mainline.git',
    revision => 'v2.2.1',
    provider => git,
}

Generally speaking, Puppet is not a good tool when you want things done in a tight time frame. Puppetlabs solution to these tasks is MCollective, which I have used and disliked.

answered May 25 '12 at 17:59

Daniel C. Sobral

5,713
6
34
48

Care to elaborate on what/why you disliked about MCollective? – François Beausoleil Apr 19 '13 at 20:29
@FrançoisBeausoleil It requires you to write ruby scripts for anything you want to do; it is a client-server application, so any failure on the servers makes them unreachable; it requires you to deploy your modules to the servers before using them; it requires a running process on the server, and, at that, a ruby one; it has an extremely unintuitive CLI, regarding server selection, modules, mcollective parameters vs modules parameters; it has very poor error reporting; the documentation sucks; the existing modules are buggy. Probably other things I'm forgetting as well. – Daniel C. Sobral Apr 19 '13 at 20:47

score 0 · Answer 4 · answered Nov 29 '17 at 06:35

To deploy Python code in a production environment you should strongly consider using virtualenv.

If you're deploying via Puppet you should look at the python-puppet module. It allows you to orchestrate everything you should need in a production Python environment, including virtualenvs.

deploying Python code with Puppet

4 Answers4