Busboy Save Stream For Use Later

Question

I'm trying to use busboy to allow clients to upload files to my Express web server.

I have the following middleware function I'm running for Express.

module.exports = (req, res, next) => {
    req.files = {};

    let busboy;
    try {
        busboy = new Busboy({
            headers: req.headers
        });
    } catch (e) {
        return next();
    }

    busboy.on("file", (fieldname, file, filename, encoding, mimetype) => {
        req.files[fieldname] = {
            file,
            filename,
            encoding,
            mimetype
        };

        // Need to call `file.resume` to consume the stream somehow (https://stackoverflow.com/a/24588458/894067)
        file.resume();
    });
    busboy.on("finish", next);

    req.pipe(busboy);
};

As you can see, I had to add file.resume(); so that the "finish" event would be triggered, and call the next function for the middleware (https://stackoverflow.com/a/24588458/894067).

The problem is, later on, when I want to consume the stream, it says readable: false. So I'm assuming the file.resume(); discards the stream and doesn't allow it to be used in the future.

I basically want to get all the uploaded files and information associated with those files, store them on the req.files object, then consume the streams later, or not consume them if I don't want to use it. That way they remain streams and don't take up much memory, until I'm ready to consume the stream and actually do something with it (or choose to discard it).

What can I use in place of file.resume(); to ensure that the "finish" event get triggers, while allowing me to use the stream later on in the lifecycle of the request (the actual app.post routes, instead of middleware)?

The client might also upload multiple files. So I need any solution to handle multiple files.

Answer 1

Would it make any sense to pipe the input stream into a PassThrough stream, like this?

const Busboy = require('busboy')
const { PassThrough } = require('stream')

const multipart = (req, res, next) => {
  req.files = new Map()
  req.fields = new Map()

  const busboy = new Busboy({ headers: req.headers })

  busboy.on('file', (fieldname, file, filename, encoding, mimetype) => {
    const stream = new PassThrough()
    file.pipe(stream)
    req.files.set(fieldname, { stream, filename, encoding, mimetype })
  })

  busboy.on(
    'field',
    (fieldname, val, fieldnameTruncated, valTruncated, encoding, mimetype) => {
      req.fields.set(fieldname, { val, encoding, mimetype })
    }
  )

  busboy.on('error', (error) => {
    next(error)
  })

  busboy.on('finish', () => {
    next()
  })

  busboy.end(req.rawBody)
}

Answer 2

If you want to handle multiple files in a single request, the procedure is a bit tricky.

Busboy goes through a single stream and fires events whenever files arrive (in sequence). You cannot get separate streams for all files at the same time with Busboy. This is not a limitation from the library, this is how HTTP works.

Your best option would be to store all files in a temporary storage, and keep information for the next middlewares with res.locals :

const Busboy = require('busboy');
const path = require('path');
const fs = require('fs');

module.exports = (req, res, next) => {
  res.locals.files = {};
  // You need to ensure the directory exists
  res.locals.someTemporaryDirectory = '/some/temp/dir/with/randomString/in/it';

  let busboy;
  try {
    busboy = new Busboy({
      headers: req.headers
    });
  } catch (e) {
    return next(e);
  }

  busboy.on("file", (fieldname, file, filename, encoding, mimetype) => {
    res.locals.files[fieldname + '_' + filename] = {
      filename,
      encoding,
      mimetype
    };
    // I skipped error handling for the sake of simplicity. Cleanup phase will be required as well 
    const tempFilePath = path.join(res.locals.someTemporaryDirectory, fieldname + '_' + filename);
    file.pipe(fs.createWriteStream(tempFilePath));
  });

  busboy.on("finish", next);

  req.pipe(busboy);
};

The next middleware shall use res.locals.someTemporaryDirectory and res.locals.files to mind their businesses (that will require a clean-up phase).

This solution may seem sub-optimal, but HTTP is like it is. You may want to issue a separate HTTP request for each file instead, but I would not recommend it as you would encounter a bunch of other issues (such as synchronization of all requests + memory management).

Whatever the solution is, it requires to get your hands dirty.