AWS S3 File Part Sizes

Question

I ran into a big issue recently. When I upload a big (let's say about 300mb) file to S3, I set the part size to be 5mb. That way I know how to calculate the checksum for when my client application downloads the file again.

When I rename the file using the S3 console, the part size changes. Total parts drop from 64 to 20 parts. Checksums fail. Clients(people) scream and flop about on the ground.

Can someone please shed some light on this issue? Is there a way to change this?

Kind Regards, EZFrag

score 1 · Accepted Answer · answered Mar 06 '18 at 15:17

In S3, there is no such thing as "renaming" a file. Instead, what's happening is S3 is:

Copying the object within S3, then
Deleting the original.

When it copies the original, it's going to copy it single chunk, or use a multi-part size that it deems appropriate. You don't have a choice in the matter.

You have 2 possible resolutions:

Option 1: If you need to rename the file, do it yourself via the AWS CLI (using s3:CopyObject). Don't use the AWS Management Console. This way you can control the part size.

Option 2: Don't use the built-in MD5/etag checksum. Instead, calculate your own and store it as custom metadata.

Would it be possible to overwrite the ETag property with our own value? — EZFrag, Mar 06 '18 at 15:26
No, the `ETag` value is calculated by S3 and you cannot change it. — Matt Houser, Mar 06 '18 at 15:27

score 0 · Answer 2 · answered Apr 29 '23 at 11:42

When you upload using the console, it uses the partsize: 17179870

I don't know if copies through console are the same but worth a try. If not, you can use the method here to figure it out.

PS. why does console use 17179870? No idea! Someones favorite number I guess.

AWS S3 File Part Sizes

2 Answers2