Node duplex stream doesn't actually emit data when read

Question

I'm working on a data-sink of sorts that is eventually to be used for Key-Value object streams in nodejs.

I've stumbled across duplex streams and started playing with them a bit to get my feet wet, but everything I try seems to not work.

At present, I've got this duplex stream:

class StorageStream extends stream.Duplex {
  constructor() {
    super({
      objectMode: true
    })

    this.values = new Map();
    this.entries = this.values.entries();
  }

  _write(data, encoding, next) {
    const { key, value } = data;

    this.values.set(key, value);

    next();
  }

  _read() {
    const next = this.entries.next();

    if (next.value) {
      this.push(next.value);
    }
  }
}

This is a SUPER CONTRIVED example, but essentially, when I write to this stream it should store the key and value in the Map, and when I read from this stream it should start reading from the map and passing them down the stream. However, this doesn't work, doing basically the below

const kvstream = createKVStreamSomeHow(); // a basic, readable stream with KV Pairs

const logger = createLoggerStreamSomeHow(); // writable stream, logs the data coming through

const storage = new StorageStream();

kvstream.pipe(storage).pipe(logger);

causes the process to just end. So I guess I'm just a bit confused as to what I'm supposed to be doing inside of the _read method.

score 1 · Answer 1 · answered Dec 03 '19 at 16:38

A couple of observations from the code provided by OP:

The iterator that loops over the keys returned by read() is generated before any keys have been set, in: this.entries = this.values.entries();. Consequently, calling read() never produce output.
If a new key is set in the Map, it won't be pushed onto the read buffer for subsequent writables to process

The duplex implementation can be simplified by using the built-in Transform (docs) constructor. The transform constructor is perfect for store-and-forward scenarios.

This is an example of how a stream Transform could be applied in this scenario. Note that the pipeline() function is not required, and has been used in this example to simplify awaiting that the readable to have emitted all its data:

const { Writable, Readable, Transform, pipeline } = require('stream');

class StorageStream extends Transform {
  constructor() {
    super({
      objectMode: true
    })

    this.values = new Map();
  }

  _transform(data, encoding, next) {
    const { key, value } = data;

    this.values.set(key, value);
    console.log(`Setting Map key ${key} := ${value}`)

    next(null, data);
  }
}

(async ()=>{
  await new Promise( resolve => {
    pipeline(
      new Readable({
        objectMode: true,
        read(){
          this.push( { key: 'foo', value: 'bar' } );
          this.push( null );
        }
      }),
      new StorageStream(),
      new Writable({
        objectMode: true,
        write( chunk, encoding, next ){
          console.log("propagated:", chunk);
          next();
        }
      }),
      (error) => {
        if( error ){
          reject( error );
        }
        else {
          resolve();
        }
      }
    );
  });
})()
  .catch( console.error );

This produces the following output

> Setting Map key foo := bar
> propagated: { key: 'foo', value: 'bar' }

And can be used as

kvstream.pipe(new StorageStream()).pipe(logger);

Node duplex stream doesn't actually emit data when read

1 Answers1