Here is my problem I'm trying to read a zipped file and do some async stuff with the data without fully reading the file's content into memory, but as soon as I do some async stuff the backpressure dissapears and before I know it the whole file is in the memory
const slowStuff = new Transform({
async transform(chunk, encoding, callback) {
this.pause();
await new Promise( (resolve) => setTimeout(resolve, 100000))
this.push(chunk);
this.resume();
callback();
},
});
const readableStream = fs.createReadStream('file.gz');
const guzipStream = zlib.createGunzip();
readableStream.pipe(guzipStream).pipe(slowStuff).pipe(more stuff...);
The file is about 8gb if I don't do anything async it only reads up parts of the file but as soon as I apply something async the whole file is in the memory.
What am I doing wrong?