Using RxJS for unknown number of consequtive HTTP Requests

Question

I need to fetch a large number of data points from our API.

These can't however all be fetched at once, as the response time would be too long, so I want to break it into multiple requests. The response looks something like this:

{
  href: www.website.com/data?skip=0&limit=150,
  nextHref: www.website.com/data?skip=150&limit=150,
  maxCount: 704,
  skip: 0,
  count: 150,
  limit:150,
  results: [...]
}

So, ultimately I need to continually call the nextHref until we actually reach the last one.

After each request, I want to take the results and concatenate them into a list of data, which will be updated on the UI.

I am relatively new to the world of Obervables but would like to create a solution with RxJS. Does anyone have an idea of how to implement this?

The part that gets me the most is that I don't know how many requests I will have to do in advance. It just needs to keep looping until it's done.

This might help you https://stackoverflow.com/a/66278720/7177558 . If this answer is not sufficient please ping me and i will create a stackblitz that matches exactly your case. — Християн Христов, Apr 19 '21 at 11:31
This should be pretty straight forward with the RxJS expand operator — Mrk Sef, Apr 19 '21 at 13:25
this can be done trivially in plain javascript using async/await — r3wt, Apr 19 '21 at 15:25

BizzyBob · Answer 1 · 2021-04-20T23:18:19.847

It looks like you can determine the number of calls to make after the first response is received. So, we can make the first call, and build an observable that returns the results of the "first call" along with the results of all subsequent calls.

We can use scan to accumulate the results into a single array.

const results$ = makeApiCall(0, 150).pipe(
  switchMap(firstResponse => {
    const pageCount = Math.ceil(firstResponse.maxCount / firstResponse.limit);
    const pageOffsets = Array(pageCount - 1).fill(0).map((_, i) => (i + 1) * firstResponse.limit);

    return concat(
      of(firstResponse),
      from(pageOffsets).pipe(
        mergeMap(offset => makeApiCall(offset, firstResponse.limit), MAX_CONCURRENT_CALLS)
      )
    );
  }),
  scan((acc, cur) => acc.concat(cur.results), [])
);

Here's a breakdown of what this does:

we first call makeApiCall() so we can determine how many other calls need made
from creates an observable that emits our array of offsets one at a time
mergeMap will execute our subsequent calls to makeApiCall() with the passed in offsets and emit the results. Notice you can provide a "concurrency" limit, to control how many calls are made at a time.
concat is used to return an observable that emits the first response, followed by the results of the subsequent calls
switchMap subscribes to this inner observable and emits the results
scan is used to accumulate the results into a single array

Here's a working StackBlitz demo.

Using RxJS for unknown number of consequtive HTTP Requests

1 Answers1