node.js server not responding while large file upload with express and busboy

Question

I am developping an file sharing platform with node.js and express.js, using busboy.

It works nicely at this time but uploading large file.

If I do, the server doesn't accept any new request while the large file is uploaded.

I that normal ? If it is, how to improve this behavior, even if that means the upload will take more time...

For now I develop and test on localhost on a pretty good PC (asus i7/8G) on ubuntu.

When I start uploadind a large file, and open a new tab to go to the app, the tab loads only when the upload is completed.

Application loading:

var app = express();

//...

app.use(bodyParser.urlencoded({
    extended: true
}));
app.use(bodyParser.json());
app.use(methodOverride());

// Request multipart parsing middleware
app.use(busboy());

// default options, immediately start reading from the request stream and 
// parsing 
app.use(busboy({ immediate: true }));

My upload method in files controller:

exports.create = function(req, res) {

    var _file = new File(req.body);

    req.busboy.on('file', function(fieldname, file, filename, encoding, mimetype) {

        file.on('data', function(data) {
            _file.data += data;
        });
        file.on('end', function() {
            _file.save(function(err) {
                if (err) {
                    console.error(err);
                    return res.status(500).send({
                        message: errorHandler.getErrorMessage(err)
                    });
                } else {
                    // doing some stuff on save
                }
            });
        });
    });

    // req.busboy.on('field', function(key, value, keyTruncated, valueTruncated) {
    //  console.log('Field [' + key + ']: value: ' + value);
    // });
    // req.busboy.on('finish', function() {
    //  console.log('Done parsing form!');
    // });

    req.pipe(req.busboy);
};

Answer 1

There's at least a few things wrong here:

• busboy is being loaded twice. You should remove the app.use(busboy({ immediate: true })); line from your app.js.
• Next, the entirety of all files are being buffered in memory (_file.data += data;). You should instead stream the file(s) somewhere, whether that's to disk, to some network storage service like Amazon's S3, or any other place outside of the process.
• This one is a moot point really once you switch to streaming, but technically with your current buffering code, you're concatenating multiple files together because the same File object is being used for the entire request. Perhaps that's not an issue if you know for sure only one file will ever be sent (for example, you are always the client), but it is worth noting.