Why does the 'finish' event fire on a Node.js transform stream before _flush is finished?

Question

I expect the 'finish' the callback to be triggered only once the callback passed to _flush has completed, but it's being triggered before _flush completes. Why?

var stream = require('stream').Transform();

// Nothing interesting here
stream._transform = function (chunk, enc, next) {
  next();
};

// _flush makes an async call.
stream._flush = function (callback) {
  console.log("Expecting 1st: Entering _flush ");

  return process.nextTick(function () {
    console.log("Expecting 2nd: Done with _flush");
    callback();
  });
};

stream.on('finish',function () {
  console.log("Expecting 3rd: Entering finish callback-- should be flushed now.");
});

stream.end("chunk");

The node.js stream documentation for finish says that the event is triggered only when all data has been flushed to the underlying system.

Looking at the source code for Node's Transform stream implementation it looks like the intent is for finish to be triggered only when the _flush callback is called:

this.once('prefinish', function() {
  if (util.isFunction(this._flush))
    this._flush(function(er) {
      done(stream, er);
    });
else
   done(stream);
});

Either I'm missed something or this is a bug with Node's Transform streams. I assume it's the former but haven't spotted my issue. Thanks for your help!

Update: Note that if I replace the call to process.nextTick() with a synchronous call to sleep, as the sleep module provides, then the issue goes away. The issue is only triggered by async calls within _flush.

Answer 1

In node v0.10, _flush() is called on 'finish', so that is more obvious to see that that is what is expected.

In master (v0.11.x) however, it looks like a new 'prefinish' event was invented and changes things a little bit. However if you look at the code, you'll see that prefinish() is called which triggers _flush(). When prefinish() returns after emitting 'prefinish', 'finish' is then emitted. It does not wait for the _flush() callback to be called before doing this.

So the resulting behavior is the same for both versions. Now whether the behavior should be changed is a separate issue. You might create a github issue about the situation and see if there is any interest in getting the behavior modified.

Answer 2

There is no direct control on the 'finish' event, and '_flush' is not part of a Writable stream.

Use a PassThrough, and do 'this.resume()' after the 'finish' event you will get a 'end' event.

Or, you could follow the pull request: https://github.com/nodejs/node-v0.x-archive/pull/7612

Or use a module: https://github.com/TomFrost/FlushWritable

Answer 3

I know this is an old question, but the documentation is rather unclear so I was still looking for an answer...

Short answer: listen to the end event instead of the finish event.

Longer answer: Although the documentation is pretty ambiguous about that (IMHO, talking about all data having been flushed to the underlying system), my understanding is that the finish event is emitted when all data from the upstream stream has been passed to your Transform, as is detected by the method _flush returning. So the "Writable" part of your Transform is finished. But that doesn't meaning your Transform is done handling it and emitting data in response!
On the other hand, the end event is emitted when there is no more data to be consumed by the downstream stream, as is detected by your _flush method calling the callback method, meaning your Transform has no more data to pass downstream, ie the "Readable" part of your Transform is done.

So the following code will provide the expected behavior:

var stream = require('stream').Transform();

// Nothing interesting here
stream._transform = function (chunk, enc, next) {
  next();
};

// _flush makes an async call.
stream._flush = function (callback) {
  console.log("Expecting 1st: Entering _flush ");

  return process.nextTick(function () {
    console.log("Expecting 2nd: Done with _flush");
    callback();
  });
};

// Here, listen on end instead of finish
stream.on('end',function () {
  console.log("Expecting 3rd: Entering finish callback-- should be flushed now.");
});

stream.end("chunk");