Why is ReadableStream preferable to Buffer when returning file data from a middleware?

Question

I have a middleware that reads multipart/form-data and returns any files submitted attached to the request body. I use busboy to process the input and return the file contents as a buffer. I read that buffers can consume a lot of memory and hence ReadableStreams are a better choice. I don't understand why this is so: even a stream has to store all the underlying data somewhere in the memory, right? So how can that be better than directly accessing the buffer itself?

Answer 1

yeah. It does store some data, but it's a small fraction of it. Say, you want to upload a file somewhere. With a buffer you would have to potentially read 1GB file and then upload the whole thing at once. If you don't have that 1GB of memory available, or if you're uploading multiple files in the same time, you will simply run out of memory. With streams, you can process data as you read it. So you load 1B of data, you upload it, you free your memory, you load another byte, you upload that etc.

If you're building a server application the idea is the same. You start receiving a file on your server and you can start processing whatever piece of the data you have before the client manages to upload the whole thing. TCP is built in such a way that the client simply cannot upload the stuff faster than you can process it. At no point will you have the entire file in your memory.