Can't upload large files to AWS with Multer S3 NodeJs

Question

I can't upload large files to aws using multer-s3. I'm using the following code:

const upload = multer({
  storage: multerS3({
    s3,
    bucket: 'welive-inc',
    acl: 'public-read',
    metadata: function (req, file, cb) {
      cb(null, {fieldName: 'TESTING_META_DATA!'});
    },
    key: function (req, file, cb) {
      cb(null, Date.now().toString() + randtoken.uid(16))
    }
  })
})

const singleUpload = upload.single('file');

router.post('/image-upload', (req, res) =>{
  singleUpload(req, res, function(err) {
    if (err) {
      console.log(err)
      return res.status(422).send({errors: [{title: 'File Upload Error', detail: err.message}] });
    }
    return res.json({'imageUrl': req.file.location});
  });
});

This code works for small files (images or really small videos), but when it comes to relatively larger files it doens't work. and the console.log(err) returns this error:

{ Error: write EPIPE
    at WriteWrap.onWriteComplete [as oncomplete] (internal/stream_base_commons.js:66:16)
  message: 'write EPIPE',
  errno: 'EPIPE',
  code: 'NetworkingError',
  syscall: 'write',
  region: 'eu-west-3',
  hostname: 'welive-inc.s3.eu-west-3.amazonaws.com',
  retryable: true,
  time: 2019-06-17T21:15:46.958Z,
  statusCode: 400,
  storageErrors: [] }

The frontend doesn't even wait for response and returns this error after couple minutes

net::ERR_EMPTY_RESPONSE

score 3 · Answer 1 · answered Sep 29 '19 at 06:38

Try overriding default upload size

var limits = {
files: 1, // allow only 1 file per request
fileSize: <Mbs allowed> * 1024 * 1024, // (replace MBs allowed with your desires)
};

then apply it to your multer instance

const upload = multer({
  limits: limits,
  storage: multerS3({
    s3,
    bucket: 'welive-inc',
    acl: 'public-read',
    metadata: function (req, file, cb) {
      cb(null, {fieldName: 'TESTING_META_DATA!'});
    },
    key: function (req, file, cb) {
      cb(null, Date.now().toString() + randtoken.uid(16))
    }
  })
})

Im uploading to s3 using multer and my issue seems to be that files over 275MB cause a err_connection_reset response from the server. Any Ideas if this is a multer issue or a an s3 issue? I tried your idea, but it didnt help. — David Tonkin, Sep 08 '20 at 09:34

Can't upload large files to AWS with Multer S3 NodeJs

1 Answers1