Node.js Readable Stream _read Usage

Question

I understand how to use writable streams in Node's new Streams2 library, but I don't understand how to use readable streams.

Take, for example, a stream wrapper around the dgram module:

var dgram = require('dgram');

var thumbs = {
  twiddle: function() {}
};

var defaults = {
  address: '0.0.0.0',
  type: 'udp4',
  port: 12345,
  broadcast: null,
  multicast: null,
  multicastTTL: 1
};

var UDPStream = function(options) {
  if (!(this instanceof UDPStream))
    return new UDPStream(options);

  Duplex.call(this);

  options = options || {};

  this.address = options.address || defaults.address;
  this.type = options.type || defaults.type;
  this.port = options.port || defaults.port;
  this.broadcast = options.broadcast || defaults.broadcast;
  this.multicast = options.multicast || defaults.multicast;
  this.multicastTTL = options.multicastTTL || defaults.multicastTTL;

  this._socket = dgram.createSocket(this.type, setup.bind(this));
  this._socket.on('message', this.push.bind(this));
};

util.inherits(UDPStream, Duplex);

var setup = function() {
  if (this.multicast) {
    this._socket.addMembership(this.multicast);
    this._socket.setMulticastTTL(this.multicastTTL);

    this.destination = this.multicast;
  } else {
    // default to using broadcast if multicast address is not specified.
    this._socket.setBroadcast(true);

    // TODO: get the default broadcast address from os.networkInterfaces() (not currently returned)
    this.destination = this.broadcast || '255.255.255.255';
  }
};

UDPStream.prototype._read = function(size) {
  thumbs.twiddle();
};

UDPStream.prototype._write = function(chunk, encoding, callback) {
  this._socket.send(chunk, 0, chunk.length, this.port, this.destination);
  callback();
};

module.exports = UDPStream;

Everything makes sense except for the _read implementation. It's literally twiddling thumbs because I don't understand what I'm supposed to do there. My data is pushed when the udp socket emits a new message, but I have no way of pausing or resuming the underlying resource. What should this look like?

Bonus question: how should `close` and `error` events be handled? — skeggse, Jul 11 '13 at 18:53
Any comments to this? I'm also confused with the implementation of readable streams when using a tcp/udp source that cannot be paused. Right now I'm simply ignoring the _read function and the highWaterMark value, and I'm also ignoring the boolean value returned by push(). If it returns false, I continue pushing more data chunks because the stream can handle it ([fromList() function](https://github.com/joyent/node/blob/master/lib/_stream_readable.js)) — Gabriel Llamas, Oct 22 '13 at 12:54

score 0 · Answer 1 · answered Jul 11 '13 at 20:47

_read is part of the pause resume mechanism. From the NodeJS API docs

When data is available, put it into the read queue by calling readable.push(chunk). If push returns false, then you should stop reading. When _read is called again, you should start pushing more data.

So in your _write function, if the socket.send call fails by either returning false or calling a callback with an error you should pause your stream. _read then can simple do this._paused = false

Might look like this.

UDPStream.prototype._read = function() {
  this._paused = false;
}

UDPStream.prototype._write = function(chunk, encoding, callback) {
  if(!this._paused)
   this._socket.send(chunk, 0, chunk.length, this.port, this.destination);
};

A duplex stream is a representation of an readable channel and an writable channel. These channels are more or less independent, and it doesn't make sense to me that the `_read` method, specified in the `Readable` "class" and therefore part of the "readable" channel, would influence the `_write` method, specified in the `Writable` "class" and therefore part of the "writable" channel. Furthermore, I suspect you're confusing `_socket.send` and `push` in your interpretation of the API docs. — skeggse, Jul 12 '13 at 03:40

score -1 · Accepted Answer · answered Oct 23 '13 at 17:56

The answer is fairly straightforward: if there is truly no way to apply backpressure to your underlying resource, your _read implementation is simply empty. The stream will take care of queueing your pushed data until it hits the highWaterMark, but guarantees nothing beyond that point. The docs say that you should "simply provide data whenever it becomes available."

Node.js Readable Stream _read Usage

2 Answers2

Linked