node.js how to handle fast producer and slow consumer with backpressure

Question

I'm very novice in node.js and don't understand the documentation about streams. Hoping to get some tips.

I'm reading a very large file line, and then for each line I'm calling an async network api.

Obviously the local file is read much faster than the async calls are completed:

var lineReader = require('readline').createInterface({
  input: require('fs').createReadStream(program.input)
});

lineReader.on('line', function (line) {
    client.execute(query, [line], function(err, result) {
        // needs to pressure the line reader here
        var myJSON = JSON.stringify(result);
        console.log("line=%s json=%s",myJSON);
    });
});

What is the way to add back pressure in the "execute" method?

Can you explain your question in brief so it could easy to understand? — Dipak, May 01 '18 at 10:29
ok so you should use promise or async await to resolve this issue and read synchronously or push data in array and then execute array synchronously. — Dipak, May 01 '18 at 10:31
There is a guide in nodejs docs about that https://nodejs.org/en/docs/guides/backpressuring-in-streams/ — Martin, May 04 '18 at 00:55

score 2 · Accepted Answer · answered May 05 '18 at 16:36

The solution is to wrap the async behavior in a stream writer and throttle the async reader from within the writer:

val count = 0;
var writable = new stream.Writable({
    write: function (line, encoding, next) {
        count++;
        if (count < concurrent) {
            next();
        }

        asyncFunctionToCall(...) {
            // completion callback
            // reduce the count and release back pressure
            count--;
            next();
            ...
      }
});

var stream = fs.createReadStream(program.input, {encoding: 'utf8'});
stream = byline.createStream(stream);
stream.pipe(writable);

hey. How does this throttles the reader? Is there a `return false;` missing in there or is it supposed to be just as is? — Ismael Abreu, Apr 11 '19 at 18:34

node.js how to handle fast producer and slow consumer with backpressure

1 Answers1