How to combine video upload chunks Node.js

Question

I'm trying to upload a large (8.3GB) video to my Node.js (Express) server by chunking using busboy. How to I receive each chunk (busboy is doing this part) and piece it together as one whole video?

I have been looking into readable and writable streams but I'm not ever getting the whole video. I keep overwriting parts of it, resulting in about 1 GB.

Here's my code:

req.busboy.on('file', (fieldname, file, filename) => {
    logger.info(`Upload of '${filename}' started`);

    const video = fs.createReadStream(path.join(`${process.cwd()}/uploads`, filename));
    const fstream = fs.createWriteStream(path.join(`${process.cwd()}/uploads`, filename));

    if (video) {
        video.pipe(fstream);
    }

    file.pipe(fstream);

    fstream.on('close', () => {
        logger.info(`Upload of '${filename}' finished`);
        res.status(200).send(`Upload of '${filename}' finished`);
    }); 
});

score 6 · Accepted Answer · answered Feb 06 '19 at 10:22

After 12+ hours, I got it figured out using pieces from this article that was given to me. I came up with this code:

//busboy is middleware on my index.js
const fs = require('fs-extra');
const streamToBuffer = require('fast-stream-to-buffer');

//API function called first
uploadVideoChunks(req, res) {
    req.pipe(req.busboy);

    req.busboy.on('file', (fieldname, file, filename, encoding, mimetype) => {
        const fileNameBase = filename.replace(/\.[^/.]+$/, '');

        //save all the chunks to a temp folder with .tmp extensions
        streamToBuffer(file, function (error, buffer) {
            const chunkDir = `${process.cwd()}/uploads/${fileNameBase}`;
            fs.outputFileSync(path.join(chunkDir, `${Date.now()}-${fileNameBase}.tmp`), buffer);
        });
    });

    req.busboy.on('finish', () => {
        res.status(200).send(`Finshed uploading chunk`);
    });
}

//API function called once all chunks are uploaded
saveToFile(req, res) {
    const { filename, profileId, movieId } = req.body;

    const uploadDir = `${process.cwd()}/uploads`;
    const fileNameBase = filename.replace(/\.[^/.]+$/, '');
    const chunkDir = `${uploadDir}/${fileNameBase}`;
    let outputFile = fs.createWriteStream(path.join(uploadDir, filename));

    fs.readdir(chunkDir, function(error, filenames) {
       if (error) {
           throw new Error('Cannot get upload chunks!');
       }

       //loop through the temp dir and write to the stream to create a new file
       filenames.forEach(function(tempName) {
           const data = fs.readFileSync(`${chunkDir}/${tempName}`);
                outputFile.write(data);
                //delete the chunk we just handled
                fs.removeSync(`${chunkDir}/${tempName}`);
           });

            outputFile.end();
        });

        outputFile.on('finish', async function () {
            //delete the temp folder once the file is written
            fs.removeSync(chunkDir);
        }
    });
}

score 2 · Answer 2 · answered Feb 06 '19 at 07:35

Use streams

multer allow you to easily handle file uploads as part of an express route. This works great for small files that don’t leave a significant memory footprint.

The problem with loading a large file into memory is that you can actually run out of memory and cause your application to crash.

use multipart/form-data request. This can be handled by assigning the readStream to that field instead in your request options

streams are extremely valuable for optimizing performance.

score 1 · Answer 3 · answered Feb 06 '19 at 10:04

Try with this code sample, I think it will work for you.

busboy.on("file", function(fieldName, file, filename, encoding, mimetype){
    const writeStream = fs.createWriteStream(writePath);
    file.pipe(writeStream);

    file.on("data", data => {
        totalSize += data.length;
        cb(totalSize);
    });

    file.on("end", () => {
        console.log("File "+ fieldName +" finished");
    });
});

You can refer this link also for resolve this problem

https://github.com/mscdex/busboy/issues/143

score 0 · Answer 4 · answered Feb 06 '19 at 05:08

0

I think multer is good with this, did you try multer?

answered Feb 06 '19 at 05:08

mustak ahmed

49
5

Yes, I tried multer. To my understanding, it does not handle chunking. – Martavis P. Feb 06 '19 at 06:17

How to combine video upload chunks Node.js

4 Answers4