2

Lately I've been having backpressure-problems with rx.js from a mongo-cursor-readable. The task is basically:

  1. Query DB A (Mongo), use its readable and convert it to an observable

  2. Perform various async transformations to the input stream, e.g.

    2.1 Query another DB B (Mongo, whatever) for additional data for an item (or a batch of items), then perform (possibly async) operations on it

    2.2 Query another DB B (Mongo, whatever) for an item (or a batch of items), then perform filtering with that additional data

  3. Count the Input stream and compare it to the output stream (as filtering can occur)

Although I do have a solution for all of the problems, I'm dealing with backpressure from the very first Mongo-DB, as 2. or 2.1/2.2 can be potentially very time intensive, but the initial query is much more performant (a.k.a consumers are slow, producers are fast)

Right now, my only solution is to limit 1. to a reasonable amount to reduce initial throughput, but this results obviously in slower throughput for the whole transformation-chain (long waiting periods for the producer until the whole chain is done working)

How do I realize a transformation chain, which is always busy, but doesn't suffer from backpressure?

I'm thinking here to reduce the speed of the producer dynamically based on the current (or projected) performance of its consumers (e.g. time taken for a transformation), but this doesn't seem very intuitive to me.

Figedi
  • 383
  • 1
  • 2
  • 11
  • If you showed some code examples, I might have a solution for you which uses RxJS's `Subject` and allows you to call `subject.next` to ask for the next set of data after you're at a certain point in the processing chain where you can accept more. – Kevin Ghadyani Nov 28 '18 at 07:00
  • hey there, yep I'm aware of the subject solution, however, this creates a hidden queue when buffering values while paused (losing values is no option) – Figedi Dec 13 '18 at 20:20
  • I don't think that's right though. Say you're reading a file. The way a Node.js stream works in this case is it waits for a block to go through the pipeline before grabbing the next one. You should be able to do the same thing by asking for a push using `subject.next`. – Kevin Ghadyani Dec 14 '18 at 00:27
  • hmm interesting, do you have an example how to wire up some rxjs pipeline with a stream such as that you can request more than one element in parallel? – Figedi Dec 22 '18 at 20:56

0 Answers0