2

I have a bunch of files on S3, and I need to download and update them via command line (BASH) on several devices.

What I do (as of now):

  1. Request the header of the file with -I and extract the ETag field
  2. Check md5sum of local file
  3. If they don't match (or file does not exist locally yet) I download the file with -f to get a reliable exit status
  4. Check $? of curl
  5. If it is 0 I compare the expected md5sum with the md5sum of the downloaded file

What I would like to do:

  1. If the file exists locally, get md5sum of it
  2. Download file with --header 'If-None-Match:<md5sum of local file>
  3. If exit status is 0 -> done!

My question: Can I rely on curls exit status? I.e. when the exit status is 0, is the downloaded file valid (assumed my file system is not corrupted in the meantime)?

Benjamin
  • 191
  • 1
  • 1
  • 5
  • 1
    I suspect you would probably be fine 99.9% of the time making this assumption, but I can imagine a few obscure cases were the request might appear to succeed, and yet you would an invalid file. One example might your session terminating in a captive-protal environment immediately before you ran the curl request. Your request would get redirected to the portal, and you would would be valid, but you would get a content you didn't expect. Of course this would only apply to http, and not https requests. – Zoredache Jun 11 '14 at 19:16

1 Answers1

0

Sorry not enough rep to comment but you may want to look at this and account for the etag not being what you expect for any file that was a multi part upload. https://stackoverflow.com/questions/6591047/etag-definition-changed-in-amazon-s3

As far as should you check the file then I would say yes if you want to be a 100% sure. Just because you may have gotten all of the correct bits over the network does not mean that it was written to your HDD correctly. Not everything is that critical that you have to check but for archival purposes I always check and you would be surprised to how many times I have to go back tweak the way files get to their final destination because on bad drives,etc.

Community
  • 1
Zero
  • 1
  • 1