0

Here is my problem I'm trying to read a zipped file and do some async stuff with the data without fully reading the file's content into memory, but as soon as I do some async stuff the backpressure dissapears and before I know it the whole file is in the memory

  const slowStuff = new Transform({
   async transform(chunk, encoding, callback) {
    this.pause(); 
    await new Promise( (resolve) => setTimeout(resolve, 100000))
    this.push(chunk);
    this.resume(); 
    callback();
  },
});

const readableStream = fs.createReadStream('file.gz');
const guzipStream = zlib.createGunzip();

readableStream.pipe(guzipStream).pipe(slowStuff).pipe(more stuff...);

The file is about 8gb if I don't do anything async it only reads up parts of the file but as soon as I apply something async the whole file is in the memory.

What am I doing wrong?

FaNaT
  • 226
  • 4
  • 15

0 Answers0