What are the semantics of the return value of nodejs stream.Readable.push()

Question

I have this code:

"use strict";

var fs = require("fs");
var stream = require('readable-stream');


var Transform = require('stream').Transform,
  util = require('util');

var TransformStream = function() {
  Transform.call(this, {objectMode: true});
};
util.inherits(TransformStream, Transform);

TransformStream.prototype._transform = function(chunk, encoding, callback) {
  if(this.push(chunk)) {
    console.log("push returned true");
  } else {
    console.log("push returned false");
  }
  callback();
};

var inStream = fs.createReadStream("in.json");
var outStream = fs.createWriteStream("out.json");
var transform = new TransformStream();

inStream.pipe(transform).pipe(outStream);

in.json is 88679467 bytes in size. The first 144 writes state that push returned true. The remaining writes (1210 of them) all state that push returned false.

out.json ends up being a full copy of in.json - so no bytes were dropped.

Which leaves me with no clue about what to do with the return value of push.

What is the right thing to do?

score 8 · Accepted Answer · edited May 23 '17 at 10:27

The push() docs (https://nodejs.org/api/stream.html#stream_readable_push_chunk_encoding) say:

return Boolean Whether or not more pushes should be performed

The purpose is "backpressure" meaning when push() returns false there is probably no space left in an outbound buffer. If your stream were reading from somewhere like a file on disk, you would stop reading when push() returns false and wait to be called again. But in your case you are implementing _transform() rather than _read(), so you don't have a lot of choice in the matter--you have received a chunk and should push() it. TransformStream will buffer any excess internally, and it can take the initiative to delay future calls to your _transform() method.

So when you implement a TransformStream you can safely ignore the return value from push().

For more on this, see:

What are the semantics of the return value of nodejs stream.Readable.push()

1 Answers1