Avoiding nesting Observables ( mergeMap on complete )

Question

Got this pseudo code for a scraping service that first explores the pagination urls then the subpage urls on the scraped "parent" website

const pagination = [["url-1", "url-2", "url-3", "url-4"], ["url-5"]];
const timeInterval = 2;

sendRequestsInIntervals(pagination, timeInterval )
  .pipe()
  .subscribe({
    next(x) {
      console.log(x);
    },
    error(err) {
      console.error(err);
    },
    complete() {
      console.log("done");
      sendRequestsInIntervals(resDataArray, timeInterval ).subscribe({
        next(x) {
          console.log(x);
        },
      });
    },
  });

I wanna avoid nesting as its an incorrect way to use observables

Is there a way to convert this to something like this:

sendRequestsInIntervals(pagination, timeInterval )
  .pipe(
    waitUntilCompletes()
    mergeMap((resDataArray) => {
      return sendRequestsInIntervals(resDataArray, timeInterval );
    })
  )
  .subscribe({
    next(x) {
      console.log(x);
    },
    error(err) {
      console.error(err);
    },
    complete() {
      console.log("done");
    },
  });

Added a pseudo function called waitUntilCompletes()

Is there such a thing in rxJS that it makes the observable in the mergeMap wait before runing until the previous observable is completed?

toArray() waits for the source observale to complete. – Mrk Sef Nov 11 '21 at 17:02 — Mrk Sef, Nov 11 '21 at 17:02
I used this approach, thanks for the tip – Lajos Bela Nov 15 '21 at 12:46 — Lajos Bela, Nov 15 '21 at 12:46

bryan60 · Answer 1 · 2021-11-11T16:25:25.877

the last operator will ignore all emissions until the final one upon completion:

sendRequestsInIntervals(pagination, timeInterval )
  .pipe(
    last(),
    mergeMap((resDataArray) => { // resDataArray is only the final emitted value from sendRequestsInIntervals() 
      return sendRequestsInIntervals(resDataArray, timeInterval );
    })
  )
  .subscribe({
    next(x) {
      console.log(x);
    },
    error(err) {
      console.error(err);
    },
    complete() {
      console.log("done");
    },
  });

or you can use reduce to gather the emissions and emit once complete:

sendRequestsInIntervals(pagination, timeInterval )
  .pipe(
    // this will gather all outter emissions into an array and emit once outter completes
    reduce((acc, val) => acc.concat(val), []),
    mergeMap((resDataArray) => { // resDataArray is all values emitted from sendRequestsInIntervals() in an array
      return sendRequestsInIntervals(resDataArray, timeInterval );
    })
  )
  .subscribe({
    next(x) {
      console.log(x);
    },
    error(err) {
      console.error(err);
    },
    complete() {
      console.log("done");
    },
  });

I'd still need the value from the other emissions not just the last one. I can do a workaround and use an array.push in the "sendRequestsInIntervals" so that the last one will contain everything but I stil wonder is this the closest available option to the idea? — Lajos Bela, Nov 11 '21 at 16:19
I realized I cant do array push, so I gotta figure out a better solution — Lajos Bela, Nov 11 '21 at 16:23
use `reduce` to gather the intermediate emissions and emit once complete — bryan60, Nov 11 '21 at 16:23

Avoiding nesting Observables ( mergeMap on complete )

1 Answers1