Handle Paging with RxJava

Question

I'm using Retrofit + RxJava on an Android app and am asking myself about how to handle the API pagination to chain calls until all data is being retrieved. Is something like this:

Observable<ApiResponse> getResults(@Query("page") int page);

The ApiResponse object has a simple structure:

class ApiResponse {
    int current;
    Integer next;
    List<ResponseObject> results;
}

The API will return a next value until is last page.

There's some good way to achieve this? Tried to combine some flatMaps(), but had no success.

Could you clarify your input and your output? A method signature with its description would be better. — zsxwing, Jan 20 '15 at 14:36

score 68 · Accepted Answer · answered Jan 23 '15 at 06:29

68

You could model it recursively:

Observable<ApiResponse> getPageAndNext(int page) {
  return getResults(page)
      .concatMap(new Func1<ApiResponse, Observable<ApiResponse>>() {

        @Override
        public Observable<ApiResponse> call(ApiResponse response) {
          // Terminal case.
          if (response.next == null) {
            return Observable.just(response);
          }
          return Observable.just(response)
              .concatWith(getPageAndNext(response.next));
        }

      });
}

Then, to consume it,

getPageAndNext(0)
    .concatMap(new Func1<ApiResponse, Observable<ResponseObject>>() {

        @Override
        public Observable<ResponseObject> call(ApiResponse response) {
          return Observable.from(response.results);
        }

    })
    .subscribe(new Action1<ResponseObject>() { /** Do something with it */ });

That should get you a stream of ResponseObject that will arrive in order, and most likely arrive in page-size chunks.

answered Jan 23 '15 at 06:29

lopar

2,432
22
14

5

Every day I discover a new RxJava operator (concatMap is new for me). Tested and approved :) – Rafael Toledo Jan 24 '15 at 16:02
1

In this example, is there a way to wait for all the results to come in and then combine the results into a single response that you can subscribe to? – Iñaqui Feb 03 '15 at 05:25
1

`toList` is probably the easiest, which will emit a single list of all elements once the `Observable` completes. there are a few other operators for grouping into collections as well, such as `reduce`. – lopar Feb 03 '15 at 05:32
To answer my own question, you can use **toList()** which conveniently returns a List of all the emitted responses. – Iñaqui Feb 05 '15 at 06:02
3

Note that I refined this a bit in another answer: http://stackoverflow.com/a/29594194/1424355 for handling cases where recursion may blow the stack. Also, it may just be a simpler alternative in general. – lopar Apr 12 '15 at 20:20
2

"where recursion may blow the stack" and it blows. @lopar I really appreciate your solution, but could you add info that this solution causes serious memory leak? It would be great if you add a link to you another answer, because I didn't notice your last comment in the first place. – Krzysztof Skrzynecki Dec 14 '15 at 10:09
i am getting stack overflow error.Is there any work around for this? – Ajinkya Sep 20 '16 at 05:49
@lopar is there any reason why you use `concatMap` in the consumer? – GVillani82 Sep 06 '18 at 17:36
As noted by other commenters, this does throw a StackOverflow exception in some cases. There's no work around for this unless you completely change the implementation of this function to use a loop instead of recursion. – Cristian Jun 30 '23 at 15:01

score 3 · Answer 2 · edited May 23 '17 at 10:30

Iopar gave a great example.

Just a small addition.
If you want to get all pages in one onNext() call.
It can be helpful when you want to zip this result with one more Observable.
You should write:

private List<String> list = new LinkedList() {
    {
        add("a");
        add("b");
        add("c");
    }
};

int count = 1;

public Observable<List<String>> getAllStrings(int c) {
    return Observable.just(list)
            .concatMap(
                    strings -> {
                        if (c == 3) {
                            return Observable.just(list);
                        } else {
                            count += 1;
                            return Observable.zip(
                                    Observable.just(list),
                                    getAllStrings(count),
                                    (strings1, strings2) -> {
                                        strings1.addAll(strings2);
                                        return strings1;
                                    }
                            );
                        }
                    }
            );
}

Usages:

getAllStrings(0)
        .subscribe(strings -> {
            Log.w(TAG, "call: " + strings);
        });

and you will get:

call: [a, b, c, a, b, c, a, b, c, a, b, c]

score 1 · Answer 3 · edited May 23 '17 at 12:02

1

I've answered my solution in a similar post: https://stackoverflow.com/a/34378263/143733

The trick or amendment to the solution provided by @Iopar is the inclusion of a 'trigger' Observable that can be emitted by a variety of ways.

In the code I posted, it is emitted once a full page of elements have been processed however it could also occur based on a user clicking a button/scrolling.

edited May 23 '17 at 12:02

Community

1
1

answered Dec 20 '15 at 05:30

Setheron

3,520
3
34
52

Handle Paging with RxJava

3 Answers3

Linked