8

My goal for file chunked uploading is the following:

  • sequential chunked uploading
  • progress update whenever a chunk is uploaded
  • retry when a chunk failed to upload
  • chunk size is 1MB
  • My enviroment is preferrably on Javascript, but can be on ASP.Net too.

I am using the AWS S3, and had configured CloudFront.

I am currently unsure which to use: the Multipart Upload API or the Chunked Upload. So what is the difference between them, and when to use them?

Harry
  • 678
  • 10
  • 26

1 Answers1

1

Here are some of the differences Ive seen so far:

Multipart Upload:

  • Supports upload in parallel
  • Only allows each part size to be a minimum of 5MB,

Chunked Upload:

Harry
  • 678
  • 10
  • 26
  • 2
    Multipart and chunked both work with sig V2 and V4. Chunked is limited to a maximum total object size of 5G. Multipart allows parallelism. – Michael - sqlbot Aug 10 '18 at 10:15
  • I've already made Multipart Upload works with v2 signature. So what do I need to do to change to Chunked Upload (preferrably using v2 signature)? – Harry Aug 10 '18 at 10:50
  • I have never encountered a use case that justified the complexity chunked uploads. They are not "chunked" in the standard sense of `Transfer-Encoding: chunked` and are completely different than multipart. – Michael - sqlbot Aug 10 '18 at 10:53
  • How about the use case where we need to display upload progress every 1MB delivered, while Multipart only allows a minium of 5MB. Also when you say these 2 are completely different, are you talking about the underlying protocol technical stuff that I as a web developer don't need to know? Can you enlighten me? – Harry Aug 10 '18 at 10:59
  • Are you using an SDK? I assumed from the question that you were writing your own implementation by hand. If an SDK, does it actually support the chunked upload protocol? I've never noticed support for that in SDKs (but then again I haven't looked). – Michael - sqlbot Aug 10 '18 at 12:12
  • No I'm not using the AWS SDK, I'm using the FineUploader on the browser, and write my own code with C# on the server. I've also noticed that the AWS SDK only supports Multipart Upload (the Javascript SDK at least) – Harry Aug 13 '18 at 03:30
  • Do part/chunk sizes need to be homogeneous (all of same size)? – darkdragon May 27 '20 at 19:12
  • 1
    @Michael-sqlbot you say multipart works with sig V4... if so, how does one avoid calculating a SHA256 hash on potentially huge single upload part of 5MB to 5GB(!). The documentation is so confusing... UploadPart docs recommends using Content-MD5 (less costly) but doesn't say how to do this with sig AWS V4. Chunk upload docs specify setting X-Amz-Content-Sha256 header to STREAMING-AWS4-HMAC-SHA256-PAYLOAD or hashing a blank space, but which headers need to be part of signature to an UploadPart PUT request? I'm glad OP is asking this question. Very confusing documentation. – AdamE Sep 23 '20 at 06:43
  • @AdamE When using POST objects in combination with sig v4, only Content-MD5 works. X-Amz-Content-Sha256 is simply ignored. How this related to multipart, I haven't tested yet. – spa900 Dec 06 '20 at 14:45
  • Hi @spa900 since writing the above, I found out the following: https://stackoverflow.com/a/64271969/796858 one can definitely construct a V4 signature without SHA-256 (as the AWS C++ SDK does). Also, the SHA-256 (if used and however "wasteful") is not ignored for content validation. You get `Status:400 : AWSCode: XAmzContentSHA256Mismatch : AWSMessage: The provided 'x-amz-content-sha256' header does not match what was computed.` whereas if Content-MD5 mismatch one gets a different message "The Content-MD5 you specified was invalid." see more details in post. – AdamE Dec 07 '20 at 19:48