Implementing backpressure in a transform stream

Question

I was reading these docs: https://nodejs.org/ja/docs/guides/backpressuring-in-streams/

It says:

Readable.pipe(Transformable).pipe(Writable);

"Backpressure will be automatically applied, but note that both the incoming and outgoing highWaterMark of the Transform stream may be manipulated and will effect the backpressure system."

I can change the highWaterMark this way:

export class JSONParser extends stream.Transform {

  constructor() {
    super({objectMode: true, highWaterMark: 3}); // 3 objects is high water mark
  }

}

but since it says that both incoming and outgoing highWaterMark of the transform stream can be manipulated - how do I change the highWaterMark of the incoming/outgoing data? My guess is that this one property controls both levels?

In essence, I am trying to slow the amount of reads in my stream, so that other I/O can be processed. The readable stream in my case is process.stdin and process.stdin is being piped to a transform stream, which I control. How can I slow the rate of reading from stdin?

score 0 · Answer 1 · answered Jul 31 '19 at 16:14

For Duplex streams (or Transform streams, which inherit from Duplex), you can set both readableHighWaterMark and writableHighWaterMark as options. Your example would just become:

export class JSONParser extends stream.Transform {

    constructor() {
        super({objectMode: true, readableHighWaterMark: 5, writableHighWaterMark: 3}); // 5 on the readable side, 3 on the writable
    }
}

Please note that if highWaterMark is set, it will override readableHighWaterMark or writableHighWaterMark (source: https://github.com/nodejs/node/blob/864860e9f3d4eed0b0b81af55197d7e525ea6306/lib/internal/streams/state.js).

Implementing backpressure in a transform stream

1 Answers1