How to properly add a child generator (sub-flow) to an Observable on RxJS?

Question

I'm using RxJS 7, and I would like to have a child generator (another Observable) emitting values based on the parent data.
I was already able to achieve this, but the solution I found is not efficient in terms of CPU usage because it needs to build a new RxJS pipeline for every parent item, and I believe I'm not using here the full potential RxJS has.

Constraints:

Emitted values from the Parent needs to be available to child generator;
Parent needs to know when child flow is done;
Child Observable can have many operators;
Efficient!

The working example:

const { from, mergeMap, reduce, lastValueFrom } = rxjs

function run() {
  const parentData = [{ parentId: 1 }, { parentId: 2 }, { parentId: 3 }]

  from(parentData)
    .pipe(mergeMap((parent) => lastValueFrom(getChildFlow(parent))))
    .subscribe((parent) => console.log(parent))
}

function getChildFlow(parent) {
  return from(childGenerator(parent))
    .pipe(reduce((acc, value) => {
      acc.inner.push(value)
      return acc
    }, { inner: [] }))
}

async function* childGenerator(parentData) {
  for await (const index of [1, 2, 3]) {
    yield { childId: index, ...parentData }
  }
}

run()

<script src="https://unpkg.com/rxjs@^7/dist/bundles/rxjs.umd.min.js"></script>

The reason I'm looking for a more efficient implementation is because it's intended for a data intensive system which can have millions of items flowing.

Questions!

Does RxJS provide some operator to cover this scenario in a more efficient implementation? I really dug RxJS's documentation and didn't found anything, but I may have missed it.
Would it be possible to reuse the flow on the above implementation? The tricky part here is that the child generator needs to have the parent data.

PS: Don't mind the implementation details of the code above, it's just an example of what I'm trying to achieve, and doesn't cover all the precautions and additional steps I have to justify the use-case.

I'm not sure I totally understand what you're after here. Is it basically one parent Observable which generates a child Observable per emission? — Will Alexander, Jan 31 '22 at 11:21
Yes, let's say I'm generating "company" entity, and for each "company", I want to fetch its "employees" and pipe to an observable which in the end of its execution will emit back to "company" flow where it can continue from there. I believe the code example can help to better understand the problem. — Pedro Kehl, Jan 31 '22 at 13:18
The code example seems unnecessarily complicated. Is there any reason you're not doing something like `parent$.pipe(mergeMap(value => child$))`? — Will Alexander, Jan 31 '22 at 14:00
Sorry, but I didn't get what you mean, because If I try "parent$.pipe(mergeMap(value => child$))", how can I have a following step on the parent flow where the data flowing is from the parent with some aggregated info from the children? Take into consideration that the implementation is design to high volume of data. (especially from the children) — Pedro Kehl, Feb 02 '22 at 00:11

score 1 · Answer 1 · answered Feb 02 '22 at 01:56

I found the solution to my problem.
It required using mergeMap, groupBy, reduce and zip. I'm not convinced it's the best solution, so if you find another approach for this that you think is more efficient, I will certainly upvote your answer and mark it as correct answer over mine.

const { from, mergeMap, tap, zip, map, groupBy, reduce } = rxjs

function run() {
  const parent$ = from([{ parentId: 1 }, { parentId: 2 }, { parentId: 3 }])
    .pipe(tap(doWhatever))

  const reducer = reduce(accumulator, [])

  const child$ = parent$
    .pipe(mergeMap(childGenerator))
    .pipe(tap(doWhatever))
    .pipe(groupBy((p) => p.parentId))
    .pipe(mergeMap((group$) => group$.pipe(reducer)))

  zip([parent$, child$])
    .pipe(map((results) => ({ ...results[0], inner: results[1] })))
    .pipe(tap(doWhatever))
    .subscribe(console.log)
}

function accumulator(acc, cur) {
  return [...acc, cur]
}

function doWhatever() {}

async function* childGenerator(parentData) {
  for await (const index of [1, 2, 3]) {
    yield { childId: index, ...parentData }
  }
}

run()

<script src="https://unpkg.com/rxjs@^7/dist/bundles/rxjs.umd.min.js"></script>

How to properly add a child generator (sub-flow) to an Observable on RxJS?

1 Answers1