How Can We Use RxJava for Paginated Web Service Calls, Where Each Page Depends on Previous Page Responses, Without Recursion?

Question

Web service APIs sometimes use pagination, where parameters to the Web service call indicate what page to retrieve. These can be roughly divided into two types:

Ones where the parameters to request a page are independent of any given page response (e.g., "give me page #3, with a page size of 10")
Ones where the parameters to request a page are dependent on some previous page response (e.g., "give me the next 10 items after the item with an identifier of foo)

This SO answer covers the first scenario nicely, where the Web service just needs a page number and all we need to determine from any given page's response is whether or not we are done.

This SO answer covers the second scenario, but it relies upon recursion, and so for large data sets we will die with a StackOverflowError.

A Relay-compatible GraphQL-powered Web service (e.g., GitHub's API) will make heavy use of the second scenario, as Relay's specification for pagination requires you to provide a "cursor" from a previous response to get the next items after that cursor position. So, I am trying to figure out a non-recursive approach for this, that still wraps everything up into a single master Observable, the way those two answers do.

you can still use "recursive" solution, augmented with some async boundary operator - e.g. `Flowable.observeOn(Schedulers.io())` - inside inner `Flowable` call — m.ostroverkhov, Aug 29 '17 at 04:39
Is web service API blocking or does it return Observable? The problem is that in order to use the API one needs to wait for the previous result to complete in order to request the next result. If you are willing to block then solution is easy. — hgrey, Aug 29 '17 at 11:43

hgrey · Accepted Answer · 2017-08-29T13:36:58.547

2

If Web Service API is blocking or you are willing to block then the solution is easy

Observable.generate(() -> new ApiResponse(page), (s, emitter) -> {
    ApiResponse r = getResults(s.next);            
    emitter.onNext(r);
    if (r.next == null) emitter.onComplete();
    return r;
});

using notation from recursive answer.

If blocking is not desirable you can use FlowableTransformers.expand from RxJava2Extensions like so

Flowable
    .just(new ApiResponse(page))
    .compose(FlowableTransformers.expand(r -> r.next == null ? Flowable.empty() : getResults(r.next)));

edited Aug 29 '17 at 13:36

answered Aug 29 '17 at 11:57

hgrey

3,033
17
21

That's an interesting point. I was stuck on using the existing `Observable`-based API, but I could switch to using a blocking call and wrap it as you illustrate. I will give that a try -- many thanks! – CommonsWare Aug 29 '17 at 12:08
Please see second part in updated answer which does not block but needs additional library – hgrey Aug 29 '17 at 13:38

score 0 · Answer 2 · answered Aug 29 '17 at 14:36

I don't know if I get it right, but I believe that you could use SyncOnSubscribe.createStateful on RxJava to solve that problem.

Check out my sample:

class SimpleTest {
  @Test
  fun testRequestTenPages() {
    getPaginatedDataFromApi()
        .take(10)
        .subscribe { println(it) }
  }

  fun apiCall(previous: Response? = null) : Response {
    return previous?.let {
      val newPage = it.page + 1
      previous.copy(id = "${it.id}_$newPage", page = newPage)
    } ?: Response("1", 1)
  }    

  fun getPaginatedDataFromApi(): Observable<Response> {
    val syncOnSubscribe = SyncOnSubscribe.createStateful<Response?, Response>(
        { null },
        { previous, observer ->
          val response = apiCall(previous)
          observer.onNext(response)
          return@createStateful response
        }
    )

    return Observable.create(syncOnSubscribe)
  }

  data class Response(val id: String, val page: Int)
}

I'm creating a stateful observable which keeps the last response as the state an use it to generate the next response.

Running this test you'll see the following output:

Response(id=1, page=1)
Response(id=1_2, page=2)
Response(id=1_2_3, page=3)
Response(id=1_2_3_4, page=4)
Response(id=1_2_3_4_5, page=5)
Response(id=1_2_3_4_5_6, page=6)
Response(id=1_2_3_4_5_6_7, page=7)
Response(id=1_2_3_4_5_6_7_8, page=8)
Response(id=1_2_3_4_5_6_7_8_9, page=9)
Response(id=1_2_3_4_5_6_7_8_9_10, page=10)

This is equivalent to Observable.generate in the other answer and similar to it also needs API calls are blocking — hgrey, Aug 29 '17 at 16:23

How Can We Use RxJava for Paginated Web Service Calls, Where Each Page Depends on Previous Page Responses, Without Recursion?

2 Answers2