1

I'm really struggling with connect-busboy and stream writing. The problem is I want to do some file validation but I can't do that without processing the data first and it seems like I have to write this to a file while I process it.

When the data comes in and I come across errors like file size or file types restrictions, I reject the request but a portion of the file is already written to disk. I really want to omit writing until the very end or perhaps stage it into a memory buffer.

This code will reject on invalid file types and size limits, but it always first creates the write stream and pipes the file into it. What I really want to do it move the "file.pipe(fstream);" into the file.on('end',... callback but that of course does not work because it has to be written as it comes in. So it seems like I need to first write it to a staging buffer, then on end flush it to a real file. Alternatively I could write to a temporary file, and on error delete that file, but that seems far more hacky and fragile. What is the best practice for this seemingly common problem?

saveFiles: function(request, options) {

return new Promise(function(resolve, reject) {

  var fstream;
  var totalData = 0;
  request.pipe(request.busboy);
  var fileIds = [];
  var assets = {'assets': {}};
  request.busboy.on('file', function(fieldname, file, originalName, encoding, contype) {
    file.on('limit', function() {
      var limitBytes = request.busboy.opts.limits.fileSize;
      var limitMB = Math.round((limitBytes / 1048576) * 100) / 100;
      return reject({errorMsg: 'Files exceeded the size limit of ' + limitMB + 'MB'});
    });
    file.on('data', function(data) {
      if (totalData === 0) {
        var type = fileType(data);
        if(type) {
          if (request.busboy.opts.limits.validFileTypes.indexOf(type.ext) === -1) {
            return reject({errorMsg: 'Invalid file type "' + type.ext + '"'}); 
          }
        }
        else {
          return reject({errorMsg: 'Invalid file type'}); 
        }
      }
      totalData += data.length;
    });
    var fileId = uuid.v1(); // used for file name on server
    var parts = originalName.split('.');
    var extension = parts[parts.length - 1].toLowerCase();
    var fileName = fileId + '.' + extension;

    fstream = fs.createWriteStream(__assets + fileName);
    file.pipe(fstream);

    file.on('end', function() {
      assets['assets'][fileId] = {
        fileName: fileId + '.' + extension,
        originalName: originalName
      };
    });

  });
  request.busboy.on('finish', function(){
    resolve(assets);
  });
});

}
mscdex
  • 104,356
  • 15
  • 192
  • 153
LMS5400
  • 492
  • 4
  • 15

0 Answers0