Uploading large size (more than 1 GB) file on Amazon S3 using Java: Large files temporary consuming lot of space in server

Question

I am trying to upload large files (more than 1 GB) on amazon S3 using Java

I am using AWS S3 multipart upload to upload large files in chunks. https://docs.aws.amazon.com/AmazonS3/latest/dev/HLuploadFileJava.html
I am using also uploading the files in chunks from the frontend.

So, the file being uploaded will be temporarily uploaded on the server in chunks and it will be uploaded on S3 in chunks. Now the problem is that this method puts a huge load on the server since this consumes server space temporarily. If multiple users are trying to upload large files at the same time then it will create an issue.

Is there any way of directly uploaded files from the user's system to amazon S3 in chunks without storing the file on server temporarily?

If upload the files via frontend directly then there a major risk of keys getting exposed.

Maybe [s3f-use](https://github.com/s3fs-fuse/s3fs-fuse) could be useful? The alternative could be to use EFS to store the files outside of the instance. — Marcin, Nov 18 '20 at 07:52
if you Implement tvm you can use temporary credentials and upload directly from the client — rupps, Nov 18 '20 at 08:11

score 1 · Accepted Answer · answered Nov 18 '20 at 08:50

1

You should leverage the upload directly from client with Signed URL There are plenty documentation for this

AWS SDK Presigned URL + Multipart upload

The presigned URLs are useful if you want your user/customer to be able to upload a specific object to your bucket, but you don't require them to have AWS security credentials or permissions.

You could also be interested in limiting the size that user is able to upload

Limit Size Of Objects While Uploading To Amazon S3 Using Pre-Signed URL

Think about signed URL as a temporary credential for client to access a specific S3 location. These credential expire in a short time so there is less security concern, but do remember to restrict the access of the signed URLs appropriately

answered Nov 18 '20 at 08:50

qkhanhpro

4,371
2
33
45

Thank you so much for this solution. This worked smoothly! – Arpan Mishra Nov 23 '20 at 06:14
How to add public-read access to all uploaded objects in the generated signed URL? – Arpan Mishra Nov 23 '20 at 07:42
https://stackoverflow.com/questions/29615828/using-s3-presigned-url-for-upload-a-file-that-will-then-have-public-read-access See if this helps – qkhanhpro Nov 23 '20 at 07:48
it for ruby, I am using Java. – Arpan Mishra Nov 23 '20 at 08:05
Lets look for how to set 'x-amz-acl' or 'acl' in the Java frameworks you are using. Sorry I can not help you on that specific part : ) – qkhanhpro Nov 23 '20 at 08:27
Adding a simple line in my generate URL request worked: generatePresignedUrlRequest.addRequestParameter("x-amz-acl", "public-read"); – Arpan Mishra Nov 23 '20 at 08:38

Uploading large size (more than 1 GB) file on Amazon S3 using Java: Large files temporary consuming lot of space in server

1 Answers1