How to load multiple rest requests

Question

I have to do multiple GET Requests to load data from an external page.

The response of the a request might return a flag that indicates that there is more data to load: "nextPage" : "/v1/catalog/products?page=2&pageSize=10",

Below is the code of my function.

I tried to implement a do while loop but I couldn't make it work. I guess there is also a smarter way to do this - maybe Switchmap?

Old Version

  loadCatalog() {
    return new Promise((resolve, reject) => {
        this.http.get<Catalog[]>(ZUORA_URL + '/v1/catalog/products?page=1&pageSize=10', { headers })
              .pipe(map(data => data))
              .subscribe(data => {
                this.catalog = data;
                resolve(true);
            });
    });
}

I want to load the complete data and store it in one place. How can I loop until there is no additional nextpage? - loading one page after another is now working but I'm still struggeling to store the responses...

Updated Version

  getProducts(url, dataSoFar = []): Observable<any[]> {
    if (!url) {
      return of (dataSoFar);
    } else {
      url = ZUORA_URL + url;
    }
    return this.http.get<any>(url, { headers }).pipe(
      switchMap(p => this.getProducts( p.nextPage, [...dataSoFar, ...p.data]))
    );
  }

  getData() {
    return this.getProducts('/v1/catalog/products');
  }

Answer 1

I am really not sure if it is recommended to wrap the observable with a Promise and update data as a side-effect.

loadCatalog(URL) {
    return new Promise((resolve, reject) => {
        this.http.get<Catalog[]>(ZUORA_URL + URL , { headers })
          .pipe(map(data => data))
          .subscribe(data => {
            resolve(data);
        });
    });
}

Now you can chain the requests to get back the data as below

async loadAllCatalogs(URL) {
  return new Promise((resolve, reject) => {
    try { 
    let catalogs = [];
    let data = await this.loadCatalog('/v1/catalog/products?page=1&pageSize=10');
    catalogs.push(data); // store catalog as an array since there may be more results based on nextPage key
    while(data.nextPage) {
       data = await this.loadCatalog(data.nextPage);
       catalogs.push(data);
    }
    resolve(catalogs);
    }
    } catch (e) {
      reject(e);
    }
  });
}

Answer 2

You can use expand to call your api recursively and reduce to reduce all the responses to a single array.

In your Service (MyService):

import { EMPTY } from 'rxjs';
import { expand, reduce, map } from 'rxjs/operators';

baseUrl = ZUORA_URL;

// Let's say you get an object like this from your API
interface ApiResponse {
  nextPage: string,
  data: any[]
}

public fetchData(apiEndpoint: string): Observable<any[]> {
  return this.http.get<ApiResponse>(baseUrl + apiEndpoint, { headers })
    .pipe(
      // recursively call the GET requests until there is no further 'nextPage' url
      expand(apiResponse => {
        if (!apiResponse.nextPage) {
          return EMPTY;
        }
        return this.http.get<ApiResponse>(apiResponse.nextPage, { headers });
      }),
      // map the api response to the data we actually want to return
      map(apiResponse => apiResponse.data),
      // reduce the data of all GET requests to a single array
      reduce((accData, data) => accData.concat(data))
    )
}

In your Component:

private products: Product[];

loadProducts() {
  this.myService.fetchData('/v1/catalog/products').subscribe(products => 
    this.products = products as Product[]
  )
}

Answer 3

Usually when things are paged, you want to only fetch things when a user actively requests them. However, I'm going to just answer your question as it stands and leave the lecture out.

I'm not sure how you would have nextPage included when you're expecting an array as your top-level in the response, so I'm going to assume the response is actually of the form:

interface CatalogResponse {
  nextPage: string | undefined;
  products: Catalog[];
}

In order to do this, you could use an Observable.

public loadProducts(url: string): Observable<Catalog[]> {
  let nextPage: Subject<string> = new Subject<string>();
  let products: Subject<Catalog[]> = new Subject<Catalog[]>();

  nextPage.subscribe((url: string) => {
    this.fetchProducts(url, products, nextPage);
  }).add(() => products.complete());

  return products;
}

private fetchProducts(url: string, products: Subject<Catalog[]>, nextPage: Subject<string>): void {
  this.http.get<CatalogResponse>(url, { headers })
          .subscribe(response => {
            products.next(response.products);
            if (response.nextPage) {
              nextPage.next(response.nextPage);
            } else {
              nextPage.complete();
            }
          });
}

You'll want to make sure you implement a "cancel" or "stop" operation in case this goes on indefinitely.