How to collect paginated API responses using spring boot WebClient?

Question

I have a paginated response from an URL, I want to keep on hitting the next page URL which I get from the previous response and keep on collecting items till I don't have a "nextPage" URL in my response. How to achieve this in a reactive way using spring boot WebClient from WebFlux with out blocking?

Request1: 

    GET /items
    response: 
    {
        items: [...]
        nextPage: "/items?page=2"
    }


    Request2: 

    GET /items?page=2
    response: 
    {
        items: [...]
        nextPage: "/items?page=3"
    }


    Request3: 

    GET /items?page=3
    response: 
    {
        items: [...]
        nextPage: null
    }

Here I have created mock urls https://karthikdivi.com/apps/paginatedReviews/withNextPageTokens/items https://karthikdivi.com/apps/paginatedReviews/withNextPageTokens/items?page=2 https://karthikdivi.com/apps/paginatedReviews/withNextPageTokens/items?page=3

How can I extract all Items from the above responses in a reactive way without blocking?

Ashish · Accepted Answer · 2018-11-19T09:39:09.930

23

Using expand, this can be achieved. Based on the mock urls provided by you.

public Mono<List<Item>> getItems() {
    String url = "https://karthikdivi.com/apps/paginatedReviews/withNextPageTokens/items";

    return fetchItems(url).expand(response -> {
        if (response.getNextPage() == null) {
            return Mono.empty();
        }
        return fetchItems(response.getNextPage());
    }).flatMap(response -> Flux.fromIterable(response.getItems())).collectList();
}

private Mono<Response> fetchItems(String url) {

         return client.get().uri(url).retrieve()
                    .bodyToMono(Response.class);
    }

edited Nov 19 '18 at 09:39

answered Nov 19 '18 at 07:57

Ashish

319
1
11

3

Returning a `Flux` would potentially allow the pipeline to continue downstream while other requests are executing. Calling `collectList` will wait until the whole expand is complete before completing the `Mono` - by wait I don’t mean “block”, just the rest of the pipeline won’t continue. – Boris the Spider Mar 23 '21 at 21:35
Why is expand better that a "simple" for loop? – Hexy Aug 15 '21 at 04:38
2

Where does the Response class come from? – nicolas-mosch Nov 13 '21 at 16:06
@Ashish It would have been better if you would have also explained about the Response Class – Jestino Sam Mar 29 '22 at 04:58
Response is just an object to which response is mapped to (one 'page'). Response class has two properties: list of items (the paginated data) and the nextPage string which is the link to the next page (if exists). – Piotr Cierpich Mar 07 '23 at 15:40

score 8 · Answer 2 · edited Nov 30 '21 at 02:29

You can achieve the desired effect using expand:

@Test
public void usingExpand(){

    Request innerData = new Request(null);
    Request middleData = new Request(innerData);
    Request rootData = new Request(middleData);

    Mono.just(rootData)
            .expand( t -> Mono.justOrEmpty(t.nextPage))
            .flatMap( t -> Flux.fromIterable(t.items))
            .subscribe(System.out::println);

}

public static class Request {
    List<String> items = new ArrayList<>();
    Request nextPage;

    public Request(Request nextPage) {
        this.items.add(UUID.randomUUID().toString());
        this.items.add(UUID.randomUUID().toString());
        this.nextPage = nextPage;
    }
}

The above code should produce the following result:

dc78317c-5552-4723-90db-5392c67655be
32ff12bb-5be1-415e-b481-dab85d9157dd
cf1e3f36-a8e2-414d-90a2-7708eeedc5be
91a6bc14-a396-483d-a66a-80bb98dc1968
c95adae3-8e6f-489b-8a9d-4cea3080e150
d6f8fe01-2c50-4574-958c-ec675331bb25

Two UUIDs from each data object.

How to collect paginated API responses using spring boot WebClient?

2 Answers2

Linked