How to fetch a remote file (e.g. from Github) in a Puppet file resource?

Question

file { 'leiningen': 
    path => '/home/vagrant/bin/lein',
    ensure => 'file',
    mode => 'a+x',
    source => 'https://raw.github.com/technomancy/leiningen/stable/bin/lein',
}

was my idea, but Puppet doesn’t know http://. Is there something about puppet:// I have missed?

Or if not, is there a way to declaratively fetch the file first and then use it as a local source?

There are a number of reasons why you probably don't want to do this. Some are documented in the feature request for HTTP(S) support in the source parameter (http://projects.puppetlabs.com/issues/5783). Basically, as the manifest is compiled every time you run the agent it would end up downloading it every time to check the contents hadn't been altered. If you only have one or two nodes this is possibly an acceptable (albeit inelegant) tradeoff, but for larger deployments it could cripple an otherwise working system. — spikeheap, Sep 17 '13 at 09:46
In Puppet 4.4+ your original example will work: https://tickets.puppetlabs.com/browse/PUP-1072 — Ajedi32, Jun 30 '16 at 18:51

score 49 · Accepted Answer · edited Dec 22 '16 at 14:06

Before Puppet 4.4, as per http://docs.puppetlabs.com/references/latest/type.html#file, the file source only accepts puppet:// or file:// URIs.

As of Puppet 4.4+, your original code would be possible.

If you're using an older version, one way to achieve what you want to do without pulling down the entire Git repository would be to use the exec resource to fetch the file.

exec{'retrieve_leiningen':
  command => "/usr/bin/wget -q https://raw.github.com/technomancy/leiningen/stable/bin/lein -O /home/vagrant/bin/lein",
  creates => "/home/vagrant/bin/lein",
}

file{'/home/vagrant/bin/lein':
  mode => 0755,
  require => Exec["retrieve_leiningen"],
}

Although the use of exec is somewhat frowned upon, it can be used effectively to create your own types. For example, you could take the snippet above and create your own resource type.

define remote_file($remote_location=undef, $mode='0644'){
  exec{"retrieve_${title}":
    command => "/usr/bin/wget -q ${remote_location} -O ${title}",
    creates => $title,
  }

  file{$title:
    mode    => $mode,
    require => Exec["retrieve_${title}"],
  }
}

remote_file{'/home/vagrant/bin/lein':
  remote_location => 'https://raw.github.com/technomancy/leiningen/stable/bin/lein',
  mode            => '0755',
}

Using "creates" means that this exec is only run if this file does not exist — rdoyle, Sep 17 '13 at 22:48
Good point, I didn't notice that. I've removed my previous comments as they're invalid. Cheers! — spikeheap, Sep 24 '13 at 13:49
I tried the github download via http:// version of the github url in puppet 5 and it still fails because it's a redirect to an https:// CDN url which puppet does not appear to handle. So, the old way is the only way in this case. You can check it on command-line: puppet resource -d file /tmp/test source=http.... — Akom, Nov 27 '17 at 17:45
a working link for a 5.5 puppet file doc is https://puppet.com/docs/puppet/5.5/types/file.html#file-attribute-source — campisano, Feb 14 '22 at 15:50

score 14 · Answer 2 · edited Apr 27 '17 at 20:49

As you're referencing a GitHub repository, I would use the Puppetlabs vcsrepo module, which would give the added benefit of being able to feed back changes or just keep up-to-date. You can install the module from the Puppet Forge using

sudo puppet module install puppetlabs/vcsrepo

Then you simply declare the repository and use file links to put the file exactly where you want it.

vcsrepo { '/opt/leiningen':
  ensure   => present,
  provider => git,
  source   => 'https://github.com/technomancy/leiningen.git',
  revision => 'stable',
}

file { "/usr/local/bin/lein": # or wherever you want the file to be
  ensure => symlink,
  target => '/opt/leiningen/bin/lein',
}

Note that the revision parameter can be used to specify the revision, tag or (as we do here) branch.

Obviously you could omit the file declaration and just update your PATH to include /opt/leiningen/bin/.

score 5 · Answer 3 · answered Dec 03 '15 at 23:35

I like the maestrodev-wget module. It can be found on Puppetlabs Forge.

Installing is simple. I use vagrant quite a lot and I have a 'bootstrap.sh' file that includes:

puppet module install maestrodev-wget

Then it's a matter of something like this:

include wget

wget::fetch { "download the jdk7 file":
  source             => 'https://a_path_to_our_internal_artifact_repo/oracle/jdk7...',
  destination        => '/tmp/jdk7...',
  timeout            => 0,
  verbose            => true,
  nocheckcertificate => true,
}

Then I use the files in my classes like normal. I added the nocheckcertificate flag because I fetch from our local https repo and I often forget the flag.

The author also makes a maven module which is quite great and also useful for fetching files from artifact repos.

score 3 · Answer 4 · answered Apr 25 '18 at 16:18

While Puppet 4.4 and newer support basic file retrieval from http sources, the built-in support is very, very basic. File retrieval is best solved using a utility type from the Puppet Forge.

There are a few options out there. A high-quality approved module that works on all platforms is lwf/remote_file. This is implemented as a native Puppet type rather than as an Exec wrapper.

Example usage:

remote_file { '/path/to/your/file':
  ensure     => present,
  source     => 'https://example.com/file.tar.gz',
  checksum   => 'd41d8cd98f00b204e9800998ecf8427e'
  proxy_host => '192.168.12.40',
  proxy_port => 3128,
}

Other features supported include passing HTTP headers controlling the SSL verification requirement, and using basic auth username/password.

score 0 · Answer 5 · answered Apr 22 '15 at 20:41

For Windows puppet users, you can use the Powershell module's and the native Powershell Invoke-WebRequest command.

exec { 'C:/home/vagrant/bin/lein':
    command => 'Invoke-WebRequest "https://raw.github.com/technomancy/leiningen/stable/bin/lein" -OutFile "C:/home/vagrant/bin/lein"',
    provider => powershell,
    creates => 'C:/home/vagrant/bin/lein'
}

How to fetch a remote file (e.g. from Github) in a Puppet file resource?

5 Answers5