Upload big file to S3 without passing through server

Question

I'm developing a web app that allows users to upload big files to the server and I'm using S3 to store those files. In order to be able to upload files of any size, I'm using s3manager.Uploader to simplify doing the multipart upload.

Because some of the files are going to be big (~15gb) I wanted to know if it's possible to do something like the presigned URLs, but for multipart uploads? I want to avoid having to upload the data to my server and then upload it to S3. Ideally I would want an URL where I can just direct a POST in a html form and avoid using my web server as a proxy for the files.

Possible duplicate of [How to do a Pre-signed POST upload to AWS S3 in Go?](http://stackoverflow.com/questions/32377782/how-to-do-a-pre-signed-post-upload-to-aws-s3-in-go) — fl0cke, Nov 09 '16 at 12:23
I suspect presigned form `POST` uploads are limited to 5GB @fl0cke. — Michael - sqlbot, Nov 09 '16 at 13:24
I don't think there's an easy answer, because amazon tends to engineer things so that the burden of coordinating things like this is placed on the user, this is how they manage to scale. You might find [this hashicorp blog post, discussing the same problem](https://www.hashicorp.com/blog/consul-s3-multipart.html) which also explains how they solved it (spoiler: with lots of code) — nothingmuch, Nov 09 '16 at 15:11
@Michael-sqlbot Are you sure about that? There are also lots of javascript libs out there that handle uploading to S3. [EvaporateJS](https://github.com/TTLabs/EvaporateJS), [FineUploader](http://fineuploader.com/), [mule-uploader](https://github.com/cinely/mule-uploader) and more — fl0cke, Nov 09 '16 at 18:44
@fl0cke If we're talking about [native form post uploads](http://docs.aws.amazon.com/AmazonS3/latest/API/sigv4-authentication-HTTPPOST.html) without another library (*"an URL where I can just direct a POST in a html form"*), then I'm reasonably confident that nothing larger is supported, though it's not easy to find something specific to `POST` uploads in the docs. Looking, not finding. — Michael - sqlbot, Nov 09 '16 at 20:25

score 1 · Accepted Answer · answered May 05 '17 at 00:18

@michael-sqlbot was right. Just doing posts there's a limit of 5GB per file. You can do a multipart upload if you presign each request before uploading, which means you need to manage the splitting of the file and asking the server to sign each request before uploading.

I ended up following @fl0cke advice and using https://github.com/TTLabs/EvaporateJS to manage the upload. It does all the file managing and you just have to implement the server side function to sign the request for each file.

score 0 · Answer 2 · edited Aug 27 '20 at 08:58

0

The limit for each object to the store is 5TB, Here you can found the reference link

edited Aug 27 '20 at 08:58

Parth Solanki

3,268
2
22
41

answered Aug 27 '20 at 07:33

Sreedhar

1

3

Is there anything which this answer contributes in addition to the older and accepted answer? – Yunnosch Aug 27 '20 at 07:38

Upload big file to S3 without passing through server

2 Answers2