44

Environment

  • I copied a file, ./barname.bin, to s3, using the command aws s3 cp ./barname.bin s3://fooname/barname.bin

  • I have a different file, ./barname.1.bin that I want to upload in place of that file

How can I upload and replace (overwrite) the file at s3://fooname/barname.bin with ./barname.1.bin?

Goals:

  • Don't change the s3 url used to access the file (new file should also be available at s3://fooname/barname.bin).
  • zero/minimum 'downtime'/unavailability of the s3 link.
Community
  • 1
  • 1
ThorSummoner
  • 16,657
  • 15
  • 135
  • 147
  • Similar question targeted at doing this in java: http://stackoverflow.com/q/9517198/1695680 – ThorSummoner May 14 '15 at 20:34
  • 2
    If you send the file to the existing key, it will overwrite that file once the upload is complete. Do you need to keep the old version? – datasage May 15 '15 at 03:33
  • Rotating / saving the old file is non critical to my question, but could be pertinent to the verbage of the question title for Google'rs. – ThorSummoner May 15 '15 at 16:32
  • 1
    S3 is based on the concept of being eventually consistent as its a distributed system. Unless the file is very large, it should replicate very quickly (within seconds). – datasage May 15 '15 at 16:40
  • Seems like your usage of the 'bazname' subdir is inconsistent in the question. – waterproof Apr 06 '16 at 19:44
  • @waterproof perhapse trying to give directory context was too much for this question, I think the objective was just to overwrite a file as atomically as possible – ThorSummoner Apr 06 '16 at 21:19
  • got it @ThorSummoner. I modified the question to clarify it and answered it with the answer I found. If you agree, can you mark my answer as correct? – waterproof Apr 20 '16 at 18:50

1 Answers1

72

As I understand it, you've got an existing file located at s3://fooname/barname.bin and you want to replace it with a new file. To replace that, you should just upload a new one on top of the old one: aws s3 cp ./barname.1.bin s3://fooname/barname.bin.

The old file will be replaced. According to the S3 docs, this is atomic, though due to EC2s replication pattern, requests for the key may still return the old file for some time.

Note (thanks @Chris Kuehl): though the replacement is technically atomic, it's possible for multipart downloads to end up with chunks from different versions of the file.

waterproof
  • 4,943
  • 5
  • 30
  • 28
  • 3
    Not directly related but this behavior really surprised me -- if you 1) start an S3 multipart upload, 2) then PUT a file (simple PUT not multipart), 3) then complete the multipart upload, the multipart upload will proclaim success but the object in your bucket is the "older" one from the simple PUT. Mentioned here: docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview.html – jamshid 52 mins ago edit – jamshid Dec 23 '16 at 03:24
  • 7
    Also note that while the replacement is technically atomic, if somebody is doing a multipart download of an object (e.g. `aws s3 cp` with default arguments) at approximately the same time as you replace it, they may receive multipart chunks from *different* versions of that file. This can lead to the downloaded file being a mix of both files (and probably very corrupted); see e.g. [this GitHub issue](https://github.com/aws/aws-cli/issues/2321). – Chris Kuehl Jan 02 '18 at 00:28
  • 1
    You might need to set the flag `--sse` if you use encryption and the bucket i versioned – Karl Pokus Dec 02 '22 at 14:13