2

All- We are working on migrating some confidential & regulatory information from Local UNIX file system to S3.

The files are copied using AWS EC2 instance into S3 using "aws s3 cp--sse aws:kms --sse-kms-key-id....... " command.

What i have noticed is the etag is different from the unix md5sum. It is exactly the same if i don't encrypt the data using kms keys.

I need to validate the upload to make sure data is not corrupt while uploading to S3, how do i validate my file is intact as etag won't match due to encryption.

Any help is really appreciated!

PS: my files are not > 5gb, i am aware of the issue with multipart upload and it is not applicable for me....

Richard
  • 381
  • 2
  • 4
  • 22

1 Answers1

7

In AWS S3 the etag is not an MD5 checksum. If just happens that this is the case in the past but AWS warns not to rely on this method for integrity checks.

In the following link is the text that I am referring to:

The ETag may or may not be an MD5 digest of the object data.

The entity tag is a hash of the object. The ETag reflects changes only to the contents of an object, not its metadata. The ETag may or may not be an MD5 digest of the object data. Whether or not it is depends on how the object was created and how it is encrypted as described below:

  • Objects created by the PUT Object, POST Object, or Copy operation, or through the AWS Management Console, and are encrypted by SSE-S3 or plaintext, have ETags that are an MD5 digest of their object data.

  • Objects created by the PUT Object, POST Object, or Copy operation, or through the AWS Management Console, and are encrypted by SSE-C or SSE-KMS, have ETags that are not an MD5 digest of their object data.

  • If an object is created by either the Multipart Upload or Part Copy operation, the ETag is not an MD5 digest, regardless of the method of encryption.

Common Response Headers

John Hanley
  • 74,467
  • 6
  • 95
  • 159
  • 1
    thanks for your response, if this is the case how do you integrity check the uploads? – Richard Dec 27 '18 at 03:26
  • 2
    Calculate the MD5 before upload. Include the header `content-md5: value`. The MD5 value is base64 encoded. AWS S3 will verify that the uploaded data matches. https://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectPUT.html – John Hanley Dec 27 '18 at 03:30