10

salt.states.file.managed takes source_hash as an argument to verify a downloaded file. This blocks me from using file.managed for a file on an online server I don't have control over. The file also changes regularly. My configuration looks like this.

download_stuff:
  file.managed:
    - name: localfile.tar.gz
    - source: http://someserver.net/onlinefile.tar.gz
    - source_hash: ???

I don't want to use cmd.run with Curl or wget because this would always download the file, even when it's already on the local machine.

I would like the know if one of the options below is possible/exists:

  • online md5 calculation service. Is there any way of getting an md5 hash of the file, using a free web service? I'm thinking of something like http://md5service.net?url={url-to-file}.
  • salt-internal conversion or workaround. Is it possible to handle this in Salt? Maybe by leaving out source_hash somehow?
  • alternative state. Is there another state in Salt for doing something like this, without losing the benefit of only downloading the file when needed?
cdMinix
  • 655
  • 1
  • 7
  • 29
  • why don't you add as a grain the current value of the hash of the downloaded package?and the next time you wish to download just compare the hash to the new package? – tudoricc Jul 20 '15 at 13:36
  • @tudoricc I want to download the latest version of ([Craft](http://buildwithcraft.com)). The link always stays the same, but it redirects to newer versions as soon as they are released. – cdMinix Jul 20 '15 at 13:45
  • @tudoricc Also thanks for asking, I added the fact that the file occasionally changes to the question. – cdMinix Jul 20 '15 at 13:52
  • Then I think you should store the hash value somewhere,grains if possible,furthermore if you want to install the package you could use the `pkg.latest` state and in the `pkgs` field for each package use the `http` keyword for specifying where to get it from – tudoricc Jul 20 '15 at 14:02
  • @tudoricc The problem is that the file isn't exactly a package, but it can be installed by essentially just extracting it and setting the right permissions. – cdMinix Jul 20 '15 at 14:08
  • @tudoricc Currently, I'm storing the hash in grains, but this isn't a permanent solution as I would have to update the hash every time a new update comes out... – cdMinix Jul 20 '15 at 14:09
  • By the looks of it you have a `.tar` file so there might be a md5 file inside of it or on the webserver[you should consider looking at this],but you can save the hash of the old file and then compare it to the hash of the new one and don't download it if that is the case.Is that the way you are doing it? – tudoricc Jul 20 '15 at 14:18
  • @tudoricc There is no md5 file inside of [the ``.tar.gz`` file](http://buildwithcraft.com/latest.tar.gz?accept_license=yes). I currently just specified the current hash of the file.Saving the hash whenever the file is downloaded sounds like a great solution, and would solve my problem. But is it possible to first save it and then call it using ``source_hash``? – cdMinix Jul 20 '15 at 14:24
  • 1
    hmmm.It's a shame they don't add the md5 of the archive,anyways...you could use some `requisites` like `Onlyif` or something else.Check this link: http://docs.saltstack.com/en/latest/ref/states/requisites.html – tudoricc Jul 20 '15 at 14:31
  • @tudoricc Would it be possible to [set a pillar in the command line](http://docs.saltstack.com/en/latest/topics/tutorials/pillar.html#setting-pillar-data-on-the-command-line) (for the calculated hash) and use it later on? I'm thinking about first using ``cmd.run`` to call ``salt '*' state.highstate pillar='{"md5": "{some-md5-calculation}"}'`` and then using the pillar in ``source_hash`` of ``file.managed``, but is it even possible to set pillars at runtime? – cdMinix Jul 20 '15 at 14:42
  • Setting pillars at runtime I really don't know you can do....but you could try this `salt '*' state.highstate pillar='{"foo": "bar"}' ` in conjuction with http://salt.readthedocs.org/en/latest/topics/tutorials/standalone_minion.html – tudoricc Jul 20 '15 at 14:53
  • Not sure if it could work for you as I don't have anyway of testing it out at the moment.but you could try to do a stateless minion – tudoricc Jul 20 '15 at 14:54
  • @cdMinx did you figure it out/> – tudoricc Jul 21 '15 at 14:29

2 Answers2

11

If you can't control the other server, please make sure that you can trust it to download its content. Not using a hash will prevent you from detecting partial or corrupted downloads. There's also no way to work with a file that has changed on the remote server.

Nevertheless you could use a state like this to circumvent the hashcode. The creates part will prevent a second download once the file has been downloaded:

bootstrap:
  cmd.run:
    - name: curl -L https://bootstrap.saltstack.com -o /etc/salt/cloud.deploy.d/bootstrap-salt.sh
    - creates: /etc/salt/cloud.deploy.d/bootstrap-salt.sh
Josh Correia
  • 3,807
  • 3
  • 33
  • 50
ahus1
  • 5,782
  • 24
  • 42
  • I don't want to use ``curl`` or ``wget``, as stated in the question, because this would always download and install/extract the file, even when it hasn't changed. – cdMinix Jul 21 '15 at 04:55
  • 2
    @cdMinix - this state will download the file exactly once. The `creates` prevents the second download. See http://docs.saltstack.com/en/latest/ref/states/all/salt.states.cmd.html for details. – ahus1 Jul 21 '15 at 05:02
2

Downloading a file with file.managed can be done since version 2016.3.0., even if you don't have access to the hash, by adding skip_verify: True. For the example given, it would be:

download_stuff:
  file.managed:
    - name: localfile.tar.gz
    - source: http://someserver.net/onlinefile.tar.gz
    - skip_verify: True

From the docs:

If True, hash verification of remote file sources (http://, https://, ftp://) will be skipped, and the source_hash argument will be ignored.

Josh Correia
  • 3,807
  • 3
  • 33
  • 50
nunop
  • 2,036
  • 2
  • 20
  • 36