Uploading Large Files on AWS S3 bucket

Question

I am trying to upload large files(PDFs, Videos) through my angular web app to AWS s3 bucket. i am using ng-file-upload for uploading file to my node server,and on node i am using multer-s3 npm package.

For small files it is working great as expected. but I am not able to upload large files using. It is not giving me any error as well. It just doesn't upload the file(if it is large), doesn't even enters the callback and gets timeout. Any way to handle uploading large files to s3 Bucket?

var uploadSingle = upload.single('uploadFile');

router.post('/uploadVideo',function(req,res,next){  
    uploadSingle(req,res,function(err){
                // doesn't come here if the file is large
            if(err){
                //Error Response , Error while uploading Module PDF;
            }
            else{
                //handling file upload
               // success response
            }
    });
}

If anyone can help me solve this issue, its greatly appreciated. If you know other libraries which can help me upload large files on AWS s3 bucket, please suggest.

yeah i tried it at first, but wasnt able to do it using aws-ask, thats why i turned to multer-s3 — Ravi Shankar Bharti, Apr 20 '17 at 12:40
@RaviShankarBharti, did you ever solve this? I am using Multer for s3 upload and am also having trouble with large files (anything over 275mb fails and gives me a ERR_CONNECTION_RESET response) — Different Gravity, Sep 08 '20 at 11:18

score 0 · Answer 1 · answered Apr 18 '20 at 07:58

You can use multipart upload API to upload large files in S3 (up to 5 TB).

From the documentation:

The multipart upload API is designed to improve the upload experience for larger objects. You can upload objects in parts. These object parts can be uploaded independently, in any order, and in parallel. You can use a multipart upload for objects from 5 MB to 5 TB in size. For more information, see Uploading Objects Using Multipart Upload API.

If you're using the SDK:

/* The following example initiates a multipart upload. */

 var params = {
  Bucket: "examplebucket", 
  Key: "largeobject"
 };
 s3.createMultipartUpload(params, function(err, data) {
   if (err) console.log(err, err.stack); // an error occurred
   else     console.log(data);           // successful response
   /*
   data = {
    Bucket: "examplebucket", 
    Key: "largeobject", 
    UploadId: "ibZBv_75gd9r8lH_gqXatLdxMVpAlj6ZQjEs.OwyF3953YdwbcQnMA2BLGn8Lx12fQNICtMw5KyteFeHw.Sjng--"
   }
   */
 });

score 0 · Answer 2 · answered Oct 05 '21 at 07:24

0

I faced the same issue you did back then but I didn't seem to find a good solution to that even now as well. So, I implemented the multipart upload process myself and to help others like me I've written an article on the same. You can find the article here. Hope you find this article useful.

answered Oct 05 '21 at 07:24

Ankan Bag

91
5

1

Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Oct 05 '21 at 07:27

Uploading Large Files on AWS S3 bucket

2 Answers2