RxJava takeWhile seems ignored. Should only execute for the first page

Question

I am trying to upload database entities in batches using workmanager. The work fires alright, however the service doing the paging doesn't work as expected.

What I would like to do:

Paginate through the database entries I need to upload (last timestamp on the server is newer than on the client). Get the page as a List
map the entries to the api objects
pass on the entities to another method which fires the API call once, sending a list of entries

Problems:

The pagination seemingly ignores the takeWhile operator and executes until it hits the maximum execution limit
The uploadEntitiesFor call is never executed

private void pageEntityAndUpload(long lastTimestamp) {
    Disposable ax = Observable.range(0, MAX_ITERATION)
            .doOnNext(integer -> {
                logInformation.logInformation(TAG, null,
                        String.format("Paging %s for user: %s, Offset (page:%s): %s, Limit: %s",
                                resourceName, userId, integer, integer * limit, limit)
                );
            })
            .concatMap(integer -> pageSupplierFunction.page(userId, lastTimestamp, integer * limit, limit))
            .doOnNext(dbEntities -> {
                logInformation.logInformation(TAG, null,
                        String.format("Found %s: %s", resourceName, dbEntities.stream()
                                .map(EntityBase::getId)
                                .collect(Collectors.joining(",")))
                );
            })
            .takeWhile(dbEntities -> !dbEntities.isEmpty())
            .flatMapIterable(a -> a)
            .map(e -> entityToApiMapper.apply(e))
            .toList()
            //.subscribeOn(Schedulers.io())
            .subscribeOn(Schedulers.single())
            .subscribe(
                    apiEntities -> uploadEntitiesFor(apiEntities),
                    throwable -> logError.logError(TAG, throwable, String.format("Failed to read and convert %s to api objects.", resourceName))
            );
}

The interfaces:

private PageSupplierFunction<DBT> pageSupplierFunction;
...
@FunctionalInterface
public interface PageSupplierFunction<T> {
    public Observable<List<T>> page(String userId, long lastTimestamp, int offset, int limit);
}

private Function<DBT, AT> entityToApiMapper;

The page supplier comes from a Room Dao, for example (it's an internal application, I need to store every users's stuff...):

@Query("SELECT * FROM partner " +
        "WHERE user_id = ....)" +
        "LIMIT :limit OFFSET :offset")
Observable<List<Partner>> pagePartnerSuggestionsFor(String userId, long lastTimestamp, int offset, int limit);

During execution, the log output:

2021-05-18 15:41:33.684 25902-26054/... I/PartnerUploadService: Paging Partner for user: TWnF0Tk....Uk1, Offset (page:0): 0, Limit: 1000
2021-05-18 15:41:34.189 25902-26054/... I/PartnerUploadService: Paging Partner for user: TWnF0Tk....Uk1, Offset (page:1): 1000, Limit: 1000
2021-05-18 15:41:34.191 25902-26054/... I/PartnerUploadService: Paging Partner for user: TWnF0Tk....Uk1, Offset (page:2): 2000, Limit: 1000
2021-05-18 15:41:34.206 25902-26054/... I/PartnerUploadService: Paging Partner for user: TWnF0Tk....Uk1, Offset (page:3): 3000, Limit: 1000
2021-05-18 15:41:34.207 25902-26054/... I/PartnerUploadService: Paging Partner for user: TWnF0Tk....Uk1, Offset (page:4): 4000, Limit: 1000
2021-05-18 15:41:34.209 25902-26054/... I/PartnerUploadService: Paging Partner for user: TWnF0Tk....Uk1, Offset (page:5): 5000, Limit: 1000
2021-05-18 15:41:34.217 25902-26054/... I/PartnerUploadService: Paging Partner for user: TWnF0Tk....Uk1, Offset (page:6): 6000, Limit: 1000
2021-05-18 15:41:34.219 25902-26054/... I/PartnerUploadService: Paging Partner for user: TWnF0Tk....Uk1, Offset (page:7): 7000, Limit: 1000
2021-05-18 15:41:34.226 25902-26054/... I/PartnerUploadService: Paging Partner for user: TWnF0Tk....Uk1, Offset (page:8): 8000, Limit: 1000
2021-05-18 15:41:34.230 25902-26054/... I/PartnerUploadService: Paging Partner for user: TWnF0Tk....Uk1, Offset (page:9): 9000, Limit: 1000
2021-05-18 15:41:34.240 25902-26054/... I/PartnerUploadService: Paging Partner for user: TWnF0Tk....Uk1, Offset (page:10): 10000, Limit: 1000
2021-05-18 15:41:34.247 25902-26054/... I/PartnerUploadService: Paging Partner for user: TWnF0Tk....Uk1, Offset (page:11): 11000, Limit: 1000
2021-05-18 15:41:34.253 25902-26054/... I/PartnerUploadService: Paging Partner for user: TWnF0Tk....Uk1, Offset (page:12): 12000, Limit: 1000
2021-05-18 15:41:34.254 25902-26054/... I/PartnerUploadService: Paging Partner for user: TWnF0Tk....Uk1, Offset (page:13): 13000, Limit: 1000
2021-05-18 15:41:34.255 25902-26054/... I/PartnerUploadService: Paging Partner for user: TWnF0Tk....Uk1, Offset (page:14): 14000, Limit: 1000
2021-05-18 15:41:34.258 25902-26054/... I/PartnerUploadService: Paging Partner for user: TWnF0Tk....Uk1, Offset (page:15): 15000, Limit: 1000
2021-05-18 15:41:34.259 25902-26054/... I/PartnerUploadService: Paging Partner for user: TWnF0Tk....Uk1, Offset (page:16): 16000, Limit: 1000
2021-05-18 15:41:34.262 25902-26054/... I/PartnerUploadService: Paging Partner for user: TWnF0Tk....Uk1, Offset (page:17): 17000, Limit: 1000
2021-05-18 15:41:34.265 25902-26054/... I/PartnerUploadService: Paging Partner for user: TWnF0Tk....Uk1, Offset (page:18): 18000, Limit: 1000
2021-05-18 15:41:34.267 25902-26054/... I/PartnerUploadService: Paging Partner for user: TWnF0Tk....Uk1, Offset (page:19): 19000, Limit: 1000
2021-05-18 15:41:34.268 25902-26054/... I/PartnerUploadService: Paging Partner for user: TWnF0Tk....Uk1, Offset (page:20): 20000, Limit: 1000
2021-05-18 15:41:34.270 25902-26054/... I/PartnerUploadService: Paging Partner for user: TWnF0Tk....Uk1, Offset (page:21): 21000, Limit: 1000
2021-05-18 15:41:34.271 25902-26054/... I/PartnerUploadService: Paging Partner for user: TWnF0Tk....Uk1, Offset (page:22): 22000, Limit: 1000
2021-05-18 15:41:34.276 25902-26054/... I/PartnerUploadService: Paging Partner for user: TWnF0Tk....Uk1, Offset (page:23): 23000, Limit: 1000
2021-05-18 15:41:34.279 25902-26054/... I/PartnerUploadService: Paging Partner for user: TWnF0Tk....Uk1, Offset (page:24): 24000, Limit: 1000
2021-05-18 15:41:34.283 25902-26054/... I/PartnerUploadService: Paging Partner for user: TWnF0Tk....Uk1, Offset (page:25): 25000, Limit: 1000
2021-05-18 15:41:34.285 25902-26054/... I/PartnerUploadService: Paging Partner for user: TWnF0Tk....Uk1, Offset (page:26): 26000, Limit: 1000
2021-05-18 15:41:34.287 25902-26054/... I/PartnerUploadService: Paging Partner for user: TWnF0Tk....Uk1, Offset (page:27): 27000, Limit: 1000
2021-05-18 15:41:34.291 25902-26054/... I/PartnerUploadService: Paging Partner for user: TWnF0Tk....Uk1, Offset (page:28): 28000, Limit: 1000
2021-05-18 15:41:34.294 25902-26054/... I/PartnerUploadService: Paging Partner for user: TWnF0Tk....Uk1, Offset (page:29): 29000, Limit: 1000
2021-05-18 15:41:34.298 25902-26054/... I/PartnerUploadService: Paging Partner for user: TWnF0Tk....Uk1, Offset (page:30): 30000, Limit: 1000
2021-05-18 15:41:34.307 25902-26054/... I/PartnerUploadService: Paging Partner for user: TWnF0Tk....Uk1, Offset (page:31): 31000, Limit: 1000
2021-05-18 15:41:34.311 25902-26054/... I/PartnerUploadService: Paging Partner for user: TWnF0Tk....Uk1, Offset (page:32): 32000, Limit: 1000
2021-05-18 15:41:34.314 25902-26054/... I/PartnerUploadService: Paging Partner for user: TWnF0Tk....Uk1, Offset (page:33): 33000, Limit: 1000
2021-05-18 15:41:34.316 25902-26054/... I/PartnerUploadService: Paging Partner for user: TWnF0Tk....Uk1, Offset (page:34): 34000, Limit: 1000
2021-05-18 15:41:34.316 25902-26058/... I/PartnerUploadService: Found Partner: 957690320dee4f7983070a1fb630f487
2021-05-18 15:41:34.317 25902-26054/... I/PartnerUploadService: Paging Partner for user: TWnF0Tk....Uk1, Offset (page:35): 35000, Limit: 1000
2021-05-18 15:41:34.318 25902-26054/... I/PartnerUploadService: Paging Partner for user: TWnF0Tk....Uk1, Offset (page:36): 36000, Limit: 1000
2021-05-18 15:41:34.319 25902-26054/... I/PartnerUploadService: Paging Partner for user: TWnF0Tk....Uk1, Offset (page:37): 37000, Limit: 1000
2021-05-18 15:41:34.321 25902-26054/... I/PartnerUploadService: Paging Partner for user: TWnF0Tk....Uk1, Offset (page:38): 38000, Limit: 1000
2021-05-18 15:41:34.322 25902-26054/... I/PartnerUploadService: Paging Partner for user: TWnF0Tk....Uk1, Offset (page:39): 39000, Limit: 1000
2021-05-18 15:41:34.335 25902-26054/... I/PartnerUploadService: Paging Partner for user: TWnF0Tk....Uk1, Offset (page:40): 40000, Limit: 1000
2021-05-18 15:41:34.337 25902-26054/... I/PartnerUploadService: Paging Partner for user: TWnF0Tk....Uk1, Offset (page:41): 41000, Limit: 1000
2021-05-18 15:41:34.342 25902-26054/... I/PartnerUploadService: Paging Partner for user: TWnF0Tk....Uk1, Offset (page:42): 42000, Limit: 1000
2021-05-18 15:41:34.343 25902-26054/... I/PartnerUploadService: Paging Partner for user: TWnF0Tk....Uk1, Offset (page:43): 43000, Limit: 1000
2021-05-18 15:41:34.346 25902-26054/... I/PartnerUploadService: Paging Partner for user: TWnF0Tk....Uk1, Offset (page:44): 44000, Limit: 1000
2021-05-18 15:41:34.359 25902-26054/... I/PartnerUploadService: Paging Partner for user: TWnF0Tk....Uk1, Offset (page:45): 45000, Limit: 1000
2021-05-18 15:41:34.361 25902-26054/... I/PartnerUploadService: Paging Partner for user: TWnF0Tk....Uk1, Offset (page:46): 46000, Limit: 1000
2021-05-18 15:41:34.362 25902-26054/... I/PartnerUploadService: Paging Partner for user: TWnF0Tk....Uk1, Offset (page:47): 47000, Limit: 1000
2021-05-18 15:41:34.363 25902-26054/... I/PartnerUploadService: Paging Partner for user: TWnF0Tk....Uk1, Offset (page:48): 48000, Limit: 1000
2021-05-18 15:41:34.369 25902-26054/... I/PartnerUploadService: Paging Partner for user: TWnF0Tk....Uk1, Offset (page:49): 49000, Limit: 1000

I only have a single partner to upload right now. The upload is not fired and the pagination still executes until it hits the limit and does not stop after the first page.

Dominik Setniewski · Accepted Answer · 2021-05-18T16:27:24.567

You provide your concatMap with observables which will be processed in order. Every next observable will be proccesed only if the previous one was completed.

The thing is, your Dao returns observable, which actually does not complete - room table will be observed unless subscriber is disposed.

In your case, you should only get only the current value from the database by changing Dao method, to return Single instead of Obervable (and also change concatMap to concatMapSingle.)

Example

I intentionally did not call onComplete on the emitter to simulate your case.

Observable.range(0, 5)
    .concatMap { id ->
        Observable.create<Int> { emitter ->
            emitter.onNext(id)
            // emitter.onComplete()
        }
    }
    .subscribe(
        { println("Next-$it") },
        { println("Error") },
        { println("Complete") }
    )

Result:

Next-0

Which can be fixed by changing Observable to Single.

Observable.range(0, 5)
    .concatMapSingle { id ->
        Single.create<Int> { emitter ->
            emitter.onSuccess(id)
        }
    }
    .subscribe(
        { println("Next-$it") },
        { println("Error") },
        { println("Complete") }
    )

Result:

Next-0
Next-1
Next-2
Next-3
Next-4
Complete

RxJava takeWhile seems ignored. Should only execute for the first page

1 Answers1