How can I enforce depth-first retrieval using rxjs expand?

Question

I'm working on an Angular 7 / Typescript / RxJS 6.3.3 project, using various RxJS Observables and related operators to process hierarchical collections (tree-like, specifically) of objects retrieved from a database via http server.

I thought to use the expand operator to create a depth-first search, and used concatMap to keep order... or so I hoped. Doesn't work.

See distilled example here: https://stackblitz.com/edit/rxjs-vf4zem

The console output from that example is:

dfs no delay: [1,8,9,22,23,24,2,4,7,3]
dfs with delay: [1,2,3,4,7,8,9,22,23,24]

(The second output line may vary depending on how much delay is added. The delay is intended to simulate data fetch from the http server.)

Given the data in the example, my desire is to consistently get the first line of output: a depth-first ordering. The key function from the example:

const dfs = (getter: Getter) => rootIds.pipe(
  concatMap(ids => from(ids)),
  expand(id =>
    getter(id).pipe(
      concatMap(children => from(children))
    )
  ),
  toArray()
);

Is there a way to enforce depth-first processing? Can expand not guarantee that, or is this just a poor means to accomplish getting hierarchical data into a flattened, depth-first array?

Answer 1

I think its a good question and I'd agree that it seems that for parallel fetch you'll need some additional data structure to compose results after fetching.

Yet, it was interesting to implement a recursive reconstruction via expand, so heres my sequential attempt:

sequentialDFS(getChildren: Getter, ids: number[]): Observable<number[]> {
  return of(ids).pipe(
    expand(([head, ...rest]) =>
      // here we have a sequence of ids
      // that we'll explore in left-to-right order,
      // e.g. [1, 17, 20] will...
      getChildren(head).pipe(
        switchMap(subChildren => {
          // ...will turn here into [2, 6, 17, 20]
          const acc = [...subChildren, ...rest ];
          return acc.length
            ? of(acc)
            : EMPTY;
        })
      )
    ),

    // collect the heads {{{
    map(([head])=>head),
    toArray()
    // }}}
  );
}

* I've a bit modified the getChildren method to return Observable<number[]> instead of Observable<number>

https://stackblitz.com/edit/rxjs-rf8d1j

This is by no means an answer to the parallel fetching. Just sharing it because it was fun.

Answer 2

So I played around a bit, did some manual recursion (as opposed to relying on expand), and came up with the following (also updated at the stackblitz link above):

class Test {
  // This doesn't maintain depth-first order; it can vary depending on timing.
  brokenDFS(getChildren: Getter, ids: number[]): Observable<number[]> {
    return of(ids).pipe(
      concatMap(ids => from(ids)),
      expand(id => getChildren(id)),
      toArray()
    );
  }

  workingDFS(getChildren: Getter, ids: number[]): Observable<number[]> {
    return from(ids).pipe(
      concatMap(id => this.parentAndChildren(getChildren, id)),
      toArray()
    );
  }

  private parentAndChildren(getChildren: Getter, id: number): Observable<number> {
    return of(id).pipe(
      concat(
        getChildren(id).pipe(
          map(child => this.parentAndChildren(getChildren, child)),
          concatAll()
        )
      ),
    );
  }

}

const getter = getChildrenWithDelay;
const rootIds = [1, 17, 20];
const test = new Test();
test.brokenDFS(getter, rootIds).subscribe(data => console.log(`Broken: ${data}`));
test.workingDFS(getter, rootIds).subscribe(data => console.log(`Working: ${data}`));

The output (noting that the "Broken" output can vary due to timing):

Broken: 1,17,20,21,2,6,18,7,13,14,3,4,19,15,16,5,8,9,10,11,12
Working: 1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21

(Also note I changed the tree values from original post such that a correct DFS is numerical array from 1 to 21).

So workingDFS works, but is much slower than brokenDFS, since every request to the http server must wait for everything before it to complete, whereas the brokenDFS version would run multiple requests simultaneously (tho not in correct order).

I don't know if I have better rxjs options here. I may have to revise my methods to pass not only the objects of interest, but also some structuring/sorting info, make all/many of the requests simultaneously, and then combine everything in the correct order after.