How to convert node readable stream to RX observable

Question

If I have a Node js stream, say for example from something like process.stdin or from fs.createReadStream, how can I convert this to be an RxJs Observable stream using RxJs5?

I see that RxJs-Node has a fromReadableStream method, but that looks like it hasn't been updated in close to a year.

so does it work or? Who cares how often it is updated if it works — smnbbrv, Jan 08 '17 at 20:29
@smnbbrv No doubt it works just fine, but it's RxJS4 and is not compatible with RxJS5. — cartant, Jan 08 '17 at 20:46
You could have a look at [the source](https://github.com/Reactive-Extensions/rx-node/blob/master/index.js#L45-L83) to see what it would take to convert it yourself - the implementation is pretty small. — cartant, Jan 08 '17 at 20:48
RxJS has changed so much over the past few versions, if anyone is reading this, try [this answer](https://stackoverflow.com/a/74925985/314114) below. — Andrew Philips, Dec 27 '22 at 05:35

Quentin Roy · Accepted Answer · 2021-03-04T18:23:23.083

For anyone looking for this, following Mark's recommendation, I adapted rx-node fromStream implementation for rxjs5.

import { Observable } from 'rxjs';

// Adapted from https://github.com/Reactive-Extensions/rx-node/blob/87589c07be626c32c842bdafa782fca5924e749c/index.js#L52
export default function fromStream(stream, finishEventName = 'end', dataEventName = 'data') {
  stream.pause();

  return new Observable((observer) => {
    function dataHandler(data) {
      observer.next(data);
    }

    function errorHandler(err) {
      observer.error(err);
    }

    function endHandler() {
      observer.complete();
    }

    stream.addListener(dataEventName, dataHandler);
    stream.addListener('error', errorHandler);
    stream.addListener(finishEventName, endHandler);

    stream.resume();

    return () => {
      stream.removeListener(dataEventName, dataHandler);
      stream.removeListener('error', errorHandler);
      stream.removeListener(finishEventName, endHandler);
    };
  }).share();
}

Note that it intrinsically breaks all back pressure functionalities of streams. Observables' are a push technology. All input chunks are going to be read and pushed to the observer as quickly as possible. Depending on your case, it might not be the best solution.

I haven't tested this since I moved on from what I was working on, but if anyone else has and it worked, I'll accept this answer :) — JuanCaicedo, Feb 11 '17 at 13:54
I am using it and so far it is working well. I have not unit tested it thought. — Quentin Roy, Feb 12 '17 at 05:00

Vladimir Popov · Answer 2 · 2021-06-15T17:11:06.663

5

Since Node v11.14.0 streams support for await https://nodejs.org/api/stream.html#stream_readable_symbol_asynciterator

it means you can pass stream to from() operator.

Under hood rxjs(v7.x.x) will call fromAsyncIterable() that will return Observable

edited Jun 15 '21 at 17:11

answered Jun 15 '21 at 15:41

Vladimir Popov

121
2
3

score 3 · Answer 3 · answered Jan 09 '17 at 18:42

The following should work for both v4 and v5 (disclaimer untested):

fromStream: function (stream, finishEventName, dataEventName) {
    stream.pause();

    finishEventName || (finishEventName = 'end');
    dataEventName || (dataEventName = 'data');

    return Observable.create(function (observer) {

      // This is the "next" event
      const data$ = Observable.fromEvent(stream, dataEventName);

      // Map this into an error event
      const error$ = Observable.fromEvent(stream, 'error')
        .flatMap(err => Observable.throw(err));

      // Shut down the stream
      const complete$ = Observable.fromEvent(stream, finishEventName);

      // Put it all together and subscribe
      const sub = data$
        .merge(error$)
        .takeUntil(complete$)
        .subscribe(observer);

      // Start the underlying node stream
      stream.resume();

      // Return a handle to destroy the stream
      return sub;
    })

    // Avoid recreating the stream on duplicate subscriptions
    .share();
  },

Dani · Answer 4 · 2021-01-25T00:43:11.670

3

The answers above will work, though the don't support backpressure. If you are trying to read a large file with createReadStream, they will read the whole thing in memory.

Here is my implementation with backpressure support: rxjs-stream

edited Jan 25 '21 at 00:43

answered Jan 24 '21 at 15:26

Dani

824
7
12

score 1 · Answer 5 · answered Jan 08 '17 at 21:07

1

The RxJs-Node implementation is RxJs4 based but can be ported to RxJs5 without much work https://github.com/Reactive-Extensions/rx-node/blob/87589c07be626c32c842bdafa782fca5924e749c/index.js#L52

answered Jan 08 '17 at 21:07

Mark van Straten

9,287
3
38
57

thanks, so it sounds like you would have to implement a conversion? – JuanCaicedo Jan 08 '17 at 21:24
It seems there is no rxjs5 version out yet so for now yes – Mark van Straten Jan 09 '17 at 06:58

score 0 · Answer 6 · answered Dec 27 '22 at 05:34

Anyone coming here recently (RxJS 7 & Node 18+) should use the following code.

Why does this work? RxJS has been updated to handle stream like objects. When you pass a ReadStream to RxJS, it tests if it is ReadableStreamLike and then turns it into an AsyncGenerator.

import { from } from 'rxjs';

const file = fs.createReadStream(fileName);

const file$ = from(file).subscribe({
  next:  (dat) => { ... },
  error: (err) => { ... },
  complete: () => { ... }
});

score 0 · Answer 7 · answered Mar 12 '23 at 12:58

Assuming you are trying to read from a csv file, the following method is the cleanest implementation that I have found to return and observable with the data parsed into objects.

In this example I used a tab sepperated file, but you can also use this approach for csv files. It uses csv-parse to map data onto the right interface.

import * as fs from 'fs';
import { parse } from 'csv-parse';
import type { Parser } from 'csv-parse';
import { Observable } from 'rxjs';

interface Columns {
  columnA: string;
  columnB: string;
}

function readTabFile(): Observable<Columns[]> {
  const parser: Parser = parse({
    delimiter: '\t',
    columns: ['columnA', 'columnB'],
  });
  return new Observable((observer) => {
    const lines: Columns[] = [];
    const stream = fs.createReadStream('./file.TAB', {
      encoding: 'utf8',
    });

    parser.on('data', (row: Columns) => {
      lines.push(row);
    });

    parser.on('end', () => {
      observer.next(lines);
      observer.complete();
    });

    stream.pipe(parser);
  });
}

How to convert node readable stream to RX observable

7 Answers7