1

I would like to convert an RxJs Observable that emits data to a nodeJS stream so I can stream this data in chunks via a nodeJS application endpoint.

Is this even possible?

Pseudo code in a nodeJS router (obviously not working):

out.post('/data', [
    body().isArray(),
], (req: Request, res: Response, next: NextFunction) => {
    const errors = validationResult(req);
    if (!errors.isEmpty()) {
        next(new RequestValidationFailedError(errors.array()[0]));
    }

    const observable = service.getData(req.context); 
    req.pipe(observable);
});
Deniss M.
  • 3,617
  • 17
  • 52
  • 100

2 Answers2

2

What about using simply the next and complete functions of an Observer, something like

service.getData(req.context).subsribe({
  next: data => res.write(data),
  complete: () => res.end()
})

UPDATE: managing clients dropping the connection

If we want to terminate the Obserable stream in case the client drops the connection, we can unsubscribe from the Observable if the event "close" is emitted by the http.Request. The logic in this case would look like this

const subscription = service.getData(req.context).subsribe({
  next: data => res.write(data),
  complete: () => res.end()
})

req.on("close", function() {
  // request closed unexpectedly
  subscription.unsubscribe();
});
Picci
  • 16,775
  • 13
  • 70
  • 113
  • this doesn't return a response once the first one becomes available. it returns the final response only if you drop the connection. – Deniss M. Oct 16 '20 at 12:31
  • 1
    According to the [node http documentation](https://nodejs.org/api/http.html#http_response_write_chunk_encoding_callback) the `write` method of `http.ServerResponse` should stream the response if called more than once (_"The first time response.write() is called, it will send the buffered header information and the first chunk of the body to the client. The second time response.write() is called, Node.js assumes data will be streamed, and sends the new data separately. That is, the response is buffered up to the first chunk of the body."_). But maybe I am missing something. – Picci Oct 16 '20 at 12:47
  • not sure. But through postman it doesn't work correctly. What's also not very good is that if the client drops the connection - the response is still being published (internally in the logs I mean) – Deniss M. Oct 16 '20 at 12:50
  • 1
    You can manage clients dropping the connections unsubscribing the subscription. I have updated my answer to reflect this. – Picci Oct 16 '20 at 16:27
  • thank you! This seem to be just what I needed. The only problem is that on the client side there is no support for json streams :D – Deniss M. Oct 17 '20 at 16:11
  • This `write` method won't do justice. We need `pipe` for this specific task. `write` waits for the stream to finish and only then returns the result to the client – Deniss M. Oct 19 '20 at 07:33
  • 1
    If I understand the [Node documentation](https://nodejs.org/api/http.html#http_response_write_chunk_encoding_callback) right, the `write` method of HttpResponse actually streams the data to the client (_"The first time response.write() is called, it will send the buffered header information and the first chunk of the body to the client. The second time response.write() is called, Node.js assumes data will be streamed, and sends the new data separately. That is, the response is buffered up to the first chunk of the body."_). – Picci Oct 19 '20 at 08:02
  • it works perfectly after I added the content type of `application/stream+json` to response :) Thanks a billion for your help, but as I mentioned the client angular application can't seem to consume pure json streams so I would need to migrate to events or websockets :) – Deniss M. Oct 19 '20 at 08:17
  • 1
    Websockets with Observables work great. You can [look at this article](https://medium.com/free-code-camp/reactive-thinking-how-to-design-a-distributed-system-with-rxjs-websockets-and-node-57d772f89260) where, among other things, an implementation of Websockets+Observables is described – Picci Oct 19 '20 at 08:26
  • thanks, but I tend more towards events if taking into account what could happen when configuring websockets on nginx. – Deniss M. Oct 19 '20 at 08:32
0

This is approximately what I use. I needed the conversion for type compliance.

As far as I know Observables don't start emitting data until subscribed. The code below subscribes right away and immediately starts pushing data.

import { Readable } from 'stream';

/**
 * Convert an observable to a readable.
 */
function convertToReadable(observable: Observable<string>): Readable => {
    const readable = new Readable({
        // https://stackoverflow.com/q/74670330/4417769
        read() { }
    });

    observable.subscribe(
        (value: string): void => {
            readable.push(value);
        },
        (error: Error): void => {
            readable.destroy(error);
        },
        (): void => {
            // complete
            readable.push(null);
        }
    );

    return readable;
};
sezanzeb
  • 816
  • 8
  • 20