RxJava2 & Retrofit: How to get pages of data

Question

Goal: I want to repeatedly call a Retrofit service (GET) that returns paged data, until I've exhausted its pages. Going from page 0 to page n.

First, I've looked at these two answers already. The first actually works, but I'm not overly fond of the recursive solution as it could lead to stack overflow. The second fails the moment you try to use a scheduler.

Here's a sample of the second:

Observable.range(0, 5/*Integer.MAX_VALUE*/) // generates page values
    .subscribeOn(Schedulers.io())           // need this to prevent UI hanging
    // gamesService uses Schedulers.io() by default
    .flatMapSingle { page -> gamesService.getGames(page) }
    .takeWhile { games -> games.isNotEmpty() } // games is a List<Game>
    .subscribe(
        { games -> db.insertAll(games) },
        { Logger.e(TAG, it, "Error getting daily games: ${it.message}") }
    )

What I expect this to do is stop the moment that gamesService.getGames(page) returns an empty list. Instead, it continues hitting the endpoint for an indeterminate number of times, with incrementing page values. I have experimented a bit in unit tests with Single.just(intVal) and determined that the problem appears to be the fact that my service is automatically subscribed on Schedulers.io(). This is how I define my Retrofit services:

private inline fun <reified T> createService(okClient: OkHttpClient): T {
    val rxAdapter = RxJava2CallAdapterFactory.createWithScheduler(Schedulers.io())
    val retrofit = Retrofit.Builder()
        .baseUrl(config.apiEndpoint.endpoint())
        .client(okClient)
        .addCallAdapterFactory(rxAdapter)
        .addConverterFactory(moshiConverterFactory())
        .build()

    return retrofit.create(T::class.java)
}

It's really not an option to not use createWithScheduler() here.

Here's another idea I tried:

val atomic = AtomicInteger(0)
Observable.generate<Int> { it.onNext(atomic.getAndIncrement()) }
    .subscribeOn(Schedulers.io())
    .flatMapSingle { page -> gamesService.getGames(page) }
    .takeWhile { games -> games.isNotEmpty() }
    .subscribe(
        { games -> dailyGamesDao.insertAll(games) },
        { Logger.e(TAG, it, "Error getting daily games: ${it.message}") }
    )

This is another case where it worked as expected right up until I introduced a Scheduler. The generator generates way too many values, when I'm expecting it to stop when the takeWhile discovers an empty list.

I've also tried various kinds of concat (concatWith, concatMap, etc).

At this point, I'm really just looking for someone to help me correct the obvious (to them) and completely basic misunderstanding I clearly have with RxJava operators.

score 2 · Answer 1 · answered May 29 '18 at 06:08

I have found a partial solution. (I may edit this answer later if and when I find my "final" solution.)

tl;dr I should convert my Singles to Observables and use the flatMap overload that takes a maxConcurrency parameter. For example:

Observable.range(0, SOME_SUFFICIENTLY_LARGE_NUMBER)
    .subscribeOn(Schedulers.io())
    .flatMap({ page -> gamesService.getGames(page).toObservable }, 1 /* maxConcurrency */)
    .takeWhile { games -> games.isNotEmpty() }
    .subscribe(
        { games -> dailyGamesDao.insertAll(games) },
        { Logger.e(TAG, it, "Error getting daily games: ${it.message}") }
    )

That basically does it. By limiting the number of concurrent threads to 1, I now have the "one after the other" behavior I was seeking. The only thing I don't like about this, and I suppose it's a minor gripe, is that my base Observable.range() can still emit a lot of values -- way more than ever get used by the downstream Singles/Observables.

PS: One reason I couldn't find this solution earlier is I was using RxJava 2.1.9. When I pushed it to 2.1.14, I had access to the new overloads. Oh well.

RxJava2 & Retrofit: How to get pages of data

1 Answers1