0

I googled that there is doesObjectExist to check if object exists but possible to check the size too?

What I want is, is the size is different then putObject to update it if the size is the same then do nothing because there's no point uploading again if there's no changes.

Thanks in advance for any advice

Dora
  • 6,776
  • 14
  • 51
  • 99
  • Yes you can. What language and which SDK are you using? You should also create a custom header with the LastUpdateTime and store that in the S3 object metadata. Then check both the LastUpdateTime and object size to determine file changes. – John Hanley Dec 24 '18 at 05:18
  • @JohnHanley using nodejs – Dora Dec 24 '18 at 05:21
  • @JohnHanley by custom header you meant meta data? – Dora Dec 24 '18 at 05:32
  • Yes, when you create or overwrite an S3 object, you can include a custom HTTP header such as `x-amz-meta-mycompany-modifiedtime` and store a timestamp that matches the original source file. This is stored in S3 as metadata just like `content-type`. It is possible to modify a file without changing the size and by checking data and object length you will have an accurate method. – John Hanley Dec 24 '18 at 06:18
  • There is even a better solution than size + last update time: Each object in S3 hash its hash stored with it you can check if the objects are really identical. – Dunedan Dec 24 '18 at 07:24
  • @Dunedan can you tell me more about it or provide example? PS, I am doing this locally so not storing anything on db – Dora Dec 24 '18 at 08:00
  • @JohnHanley thx, I kinda get the idea of it. Let me test it out – Dora Dec 24 '18 at 08:14
  • I think what you really might want is to try to upload and only upload if the objects differ. The following answer details how that works: https://stackoverflow.com/a/1775831/4779904 – Dunedan Dec 24 '18 at 09:41
  • @Dunedan - Using the etag header for md5 is not reliable today. That answer is outdated. – John Hanley Dec 24 '18 at 19:35
  • @JohnHanley I was trying to compare the `modified time` and `size` that I saved into the `meta` but I modified date + size wouldn't be too accurate. Even I tried pulling binary from s3 and compare with binary local without editing or saving, the binary is still different. Is there a better accurate way to compare? – Dora Dec 27 '18 at 23:30
  • What date are you checking? Your own custom date header, which can be accurate to milliseconds or the Cloud Storage header? Create a new question, show your code and ping me with the new question link. – John Hanley Dec 27 '18 at 23:32
  • @JohnHanley I have just posted here https://stackoverflow.com/questions/53952559/accurate-method-to-compare-the-s3-object-and-local-object – Dora Dec 28 '18 at 00:59

0 Answers0