Rxjs stream of arrays to a single value and repeat at the end of the stream

Question

I have an observable that fetches an array of items (32 each time) from an API and emits a new response until there are no items left to fetch.

I want to process said list of items one by one as soon as i get the first batch until im done with ALL items fetched.

When i'm done with the complete list, i want to repeat the process indefinitely.

Here's what i have so far:

_dataService
      .getItemsObservable()
      .pipe(
        switchMap((items) => {
          const itemList = items.map((i) => i.itemId);
          return of(itemList);
        }),
        concatMap((item) =>
            from(item).pipe(
              concatMap((item) => {
                   // do something here
                }
              )
            )
         ),
         repeat()
      ).subscribe()

Any idea on what can i do? Right now what happens is it will loop over the first batch of items and ignore the rest

score 0 · Answer 1 · answered Jan 26 '23 at 05:22

Replay wont call the service again, it will reuse the original values. Try switchMap from a behaviour subject and make it emit after you have processed the values. Really not sure why you would turn each item into an observable to concatMap. Just process the items after they are emitted.

const { of, BehaviorSubject, switchMap, delay } = rxjs;

const _dataService = {
  getItemsObservable: () => of(Array.from({ length: 32 }, () => Math.random()))
};

const bs$ = new BehaviorSubject();

bs$.pipe(
  delay(1000), // Behavior subject are synchronous will cause a stack overflow 
  switchMap(() => _dataService.getItemsObservable())
).subscribe(values => {
  values.forEach(val => {
    console.log('Doing stuff with ' + val);
  });
  bs$.next();
});

<script src="https://cdnjs.cloudflare.com/ajax/libs/rxjs/7.8.0/rxjs.umd.min.js" integrity="sha512-v0/YVjBcbjLN6scjmmJN+h86koeB7JhY4/2YeyA5l+rTdtKLv0VbDBNJ32rxJpsaW1QGMd1Z16lsLOSGI38Rbg==" crossorigin="anonymous" referrerpolicy="no-referrer"></script>

score 0 · Answer 2 · answered Jan 26 '23 at 17:50

I have an observable that fetches an array of items (32 each time) from an API and emits a new response until there are no items left to fetch.

Okay, I assume that is _dataService.getItemsObservable()?

I want to process said list of items

What does this mean? Process how? Lets assume you have some function called processItemById that processes an itemId and returns the processed item.

one by one as soon as i get the first batch until im done with ALL items fetched.

Sounds like you're turning an Observable<T[]> into an Observable<T>. You can use mergeMap (don't care about order) or concatMap (maintain order) to do this. Since you're just flattening an inner array, they'll be the same in this case.

_dataService.getItemsObservable().pipe(
  mergeMap(v => v),
  map(item => processItemById(item.itemId)),
  // Repeat here doesn't call `getItemsObservable()` again,
  // instead it re-subscribes to the observable that was returned.
  // Hopefully that's what you're counting on. It's not clear to me
  repeat()
).subscribe(processedItemOutput => {
  // Do something with the output?
});

Any idea on what can i do?

From your explanation and code, it's not clear what you're trying to do. Maybe this helps.

Right now what happens is it will loop over the first batch of items and ignore the rest

This could happen for a number of reasons.

Tip 1

Using higher-order mapping operators with RxJS::of is a code smell. Just use a regular map instead.

for example:

concatMap(v => of(fn(v)))
// or
switchMap(v => of(fn(v)))

are the same as:

map(v => fn(v))

Tip 2

I have no idea if this would help you but you can generate a new observable on each subscribe by using the delay operator.

For example:

defer(() => _dataService.getItemsObservable()).pipe(
  mergeMap(v => v),
  map(item => processItemById(item.itemId)),
  repeat()
).subscribe(processedItemOutput => {
  // Do something with the output?
});

score 0 · Answer 3 · answered Jan 27 '23 at 11:56

It looks like you want to get all records from an API that is paginated but won't tell you how many pages are there which sounds like you're looking for expand() operator which is great for recursive calls.

import { of, EMPTY, expand, range, toArray, mergeMap, concat, map, takeLast } from 'rxjs';

const MAX_PAGES = 3;
const PER_PAGE = 32;

const fetchList = (offset: number) => {
  return Math.ceil(offset / PER_PAGE) >= MAX_PAGES ? of([]) : range(offset, 32).pipe(toArray());
};

const fetchDetail = (id: number) => {
  return of(`response for ${id}`);
};

of([]) // Seed for `expand()`.
  .pipe(
    expand((acc, index) => fetchList(acc.length).pipe(
      mergeMap(list => { // Process each response array
        console.log(`Response #${index}`, list);

        // When the reponse is an empty we can stop recursive calls.
        if (list.length === 0) {
          return EMPTY;
        }

        // Process the response and make `fetchDetail()` call for each item.
        // `concat()` will guarantee order and process items one by one.
        return concat(...list.map(id => fetchDetail(id)))
          .pipe(
            toArray(),
            map(details => [...acc, ...details]), // Append the processed response to a single large array.
          );
      }),
    )),
    takeLast(1), // Only take the final array after all pages have been fetched.
  )
  .subscribe(console.log);

Working demo: https://stackblitz.com/edit/rxjs-yqd4kx?devtoolsheight=60&file=index.ts

Rxjs stream of arrays to a single value and repeat at the end of the stream

3 Answers3

Tip 1

Tip 2