Fineuploader: large files to Amazon S3

Question

For small files (smaller than about 5MB) Fineuploader sends a complete policy objet for signing, and upload succeeds including the posprocessing which is sending a temporary link to the client, and some server-side administrative stuff. I use the official FineUploader S3 demo PHP code for server. https://github.com/Widen/fine-uploader-server/blob/master/php/s3/s3demo-thumbnails.php

Example of signing request (formatted, anonimized):

{
    "expiration": "2013-12-28T12:57:42.354Z",
    "conditions": [
        {"acl": "private"},
        {"bucket": "TOP-SECRET-BUCKET-NAME"},
        {"Content-Type": "application/pdf"},
        {"success_action_status": "200"},
        {"key": "bfccb67e-5343-4e01-97ff-2dcffe681da0.pdf"},
        {"x-amz-meta-qqfilename": "plakat_a3.pdf"},
        ["content-length-range","0","1000111000111"]
    ]
}

Response contains policy and signature (formatted):

{
    "policy":"eyJleHBpcmF0aW9uIjoiMjAxMy0xMi0yOFQxMzoxODoyNy4yODhaIiwiY29uZGl0aW9ucyI6W3siYWNsIjoicHJpdmF0ZSJ9LHsiYnVja2V0IjoiZGVtb2Nza2EifSx7IkNvbnRlbnQtVHlwZSI6ImFwcGxpY2F0aW9uXC9wZGYifSx7InN1Y2Nlc3NfYWN0aW9uX3N0YXR1cyI6IjIwMCJ9LHsia2V5IjoiMWZiMjZmMjAtNjg2Ni00YjU1LTg3YTctZWZlMjNiOWMwZmY1LnBkZiJ9LHsieC1hbXotbWV0YS1xcWZpbGVuYW1lIjoicGxha2F0X2EzLnBkZiJ9LFsiY29udGVudC1sZW5ndGgtcmFuZ2UiLCIwIiwiMTAwMDExMTAwMDExMSJdXX0=",
    "signature":"Wlw1QJjwmsASyQemUWrYuktiQwE="
}

... but for larger files Fineupload sends a different signing request, which only contains headers (formatted, anonimized):

{
    "headers": "POST\n\nvideo/quicktime\n\nx-amz-acl:private\nx-amz-date:Sat, 28 Dec 2013 12:53:13 GMT\nx-amz-meta-qqfilename:20MB_stopmot-minta.mov\n/TOP_SECRET-BUCKET-NAME/d2033a4c-1e55-49a0-8589-9b1725dcd013.mov?uploads"
}

Response contains only signature (formatted):

{
    "signature":"jYtFC91wIPkZj31W\/vwuK9ClawU="
}

In that case the OPTIONS request to amazon S3 fails:

Request Headers 14:09:09.000
User-Agent: Mozilla/5.0 (Windows NT 6.1; WOW64; rv:26.0) Gecko/20100101 Firefox/26.0
Pragma: no-cache
Origin: http://top-secret.example.com
Host: TOP-SECRET-BUCKET-NAME.s3.amazonaws.com
Connection: keep-alive
Cache-Control: no-cache
Access-Control-Request-Method: POST
Access-Control-Request-Headers: authorization,content-type,x-amz-acl,x-amz-date,x-amz-meta-qqfilename
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8

Response:

Status: 403
Body: empty

Response headers:

x-amz-request-id: 625913F399C626A2
x-amz-id-2: +enheIuZT0RJ+11THF1TuNsA6bCqE4a2ppsklK84k4KXdNX4UsuGhxPf00Eb137G
Transfer-Encoding: chunked
Server: AmazonS3
Date: Sat, 28 Dec 2013 13:18:01 GMT
Content-Type: application/xml

Test cases:

Tested with smaller .mov, .mp3, .pdf, .log files: course #1 (success)
Tested with larger files (same types): course #2 (fail)

What can cause the difference? How can / should I fix it?

Have a look at the documentation: http://docs.fineuploader.com/endpoint_handlers/amazon-s3.html. Files larger than 2 MB are chunked. Fine Uploader uses S3's multipart upload API for this. If you want to disable chunking (there is no good reason to disable this), you can do so via the chunking option. — Ray Nicholus, Dec 28 '13 at 15:04
I do need large file upload in a chunked manner. I found the mistake I made, I will share it as an answer, so others will be able to do it more easily :-) — Notinlist, Dec 28 '13 at 15:46

score 4 · Accepted Answer · answered Dec 28 '13 at 15:47

I needed to add <ExposeHeader>ETag</ExposeHeader> to my CORS configuration at Amazon S3 bucket permissions. So finally it looks similar to:

<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
    <CORSRule>
        <AllowedOrigin>*</AllowedOrigin>
        <AllowedMethod>POST</AllowedMethod>
        <AllowedMethod>PUT</AllowedMethod>
        <AllowedMethod>DELETE</AllowedMethod>
        <MaxAgeSeconds>3000</MaxAgeSeconds>
        <ExposeHeader>ETag</ExposeHeader>
        <AllowedHeader>*</AllowedHeader>
    </CORSRule>
</CORSConfiguration>

Reference:

http://docs.fineuploader.com/endpoint_handlers/amazon-s3.html

Fineuploader: large files to Amazon S3

1 Answers1