Firebase and busboy, do I get the whole file in memory, or?

Question

I want to upload rather large files in a node.js/Firebase project. I am trying to understand if Busboy fits my need.

The example in the Busboy documentation ends with

req.pipe(busboy);

However this does not work in Firebase. It looks like you instead should use

busboy.end(req.rawBody);

That works for me (at least locally, under firebase serve. However there is a difference, perhaps. In the example I mentioned above you have this:

if (req.method === 'POST') {
    let busboy = new Busboy({ headers: req.headers });
    busboy.on('file', (fieldname, file, filename, encoding, mimetype) => {
      console.log('File [' + fieldname + ']: filename: ' + filename + ', encoding: ' + encoding + ', mimetype: ' + mimetype);
      file.on('data', (data) => {
        console.log('File [' + fieldname + '] got ' + data.length + ' bytes');
      });

I expected to see several outputs from `file.on("data", ...)`, but there is just one line for a 100 MB file. Does this mean that the whole file is kept in memory?

EDIT: The busboy.end(req.rawBody) is from https://cloud.google.com/functions/docs/writing/http#multipart_data

score 1 · Answer 1 · answered May 11 '19 at 16:05

1

The entire request and response for Cloud Functions input and output is kept entirely in memory before being transferred to and from the client. The maximum payload size for requests and responses is 10MB, as shown in the documentation.

When working with large files, you should consider uploading to Cloud Functions, then trigger a function from there.

answered May 11 '19 at 16:05

Doug Stevenson

297,357
32
422
441

Thanks Doug. In these case my idea was to stream the file through Busboy with a pipe to Google Cloud Storage. I am very unsure what works how in cases like this. The example in the Busboy documentation looks like you could stream a file that way. Am I misunderstanding that example? Or is it just that this does not work with Firebase yet? – Leo May 11 '19 at 22:47
1

As I said, Cloud Functions doesn't actually stream the input. It is all in memory the moment the function is invoked. Streaming doesn't scale in the way that Cloud Functions requires. You can stream from Cloud Storage using its node API, however. – Doug Stevenson May 11 '19 at 23:54
This was just another try to find a way to upload big files to GCS. Please see my other question here where you also answered: https://stackoverflow.com/questions/56237102/using-websocket-stream-to-upload-file-to-cloud-storage-with-firebase-node-js?noredirect=1#comment99103262_56237102 – Leo May 21 '19 at 21:33

Firebase and busboy, do I get the whole file in memory, or?

I expected to see several outputs from file.on("data", ...), but there is just one line for a 100 MB file. Does this mean that the whole file is kept in memory?

1 Answers1

I expected to see several outputs from `file.on("data", ...)`, but there is just one line for a 100 MB file. Does this mean that the whole file is kept in memory?