Paging 3 - How to handle old APPEND RemoteMediator requests that come trough since beta01?

Question

Since version beta01 of Paging 3, when refreshing the PagingData from a RemoteMediator, it sometimes happens that the old APPEND request from the previous generation is still executed after the refresh has completed. This seems to be expected behavior reading from this commit. When this happens, the old APPEND request calls the RemoteMediator's load method but with an outdated PagingState. This outdated PagingState can cause bugs and crashes if we are using the information in it in the load function (for example, the code snippet below uses lastItemOrNull to find the RemoteKeys for an item in the database). This breaking change (which also breaks the corresponding codelab) is not mentioned in the release notes at all. How are we supposed to handle this?

Here is an example of a RemoteMediator that breaks with beta01. The getRemoteKeyForLastItem method can return null (because the old PagingState is looking for a database entry that was deleted in the previous REFRESH) causing the InvalidObjectException to be thrown.

private const val GITHUB_STARTING_PAGE_INDEX = 1

@OptIn(ExperimentalPagingApi::class)
class GithubRemoteMediator(
        private val query: String,
        private val service: GithubService,
        private val repoDatabase: RepoDatabase
) : RemoteMediator<Int, Repo>() {

    override suspend fun load(loadType: LoadType, state: PagingState<Int, Repo>): MediatorResult {

        val page = when (loadType) {
            LoadType.REFRESH -> GITHUB_STARTING_PAGE_INDEX
            LoadType.PREPEND -> return MediatorResult.Success(true)
            LoadType.APPEND -> {
                // this can run with an outdated PagingState from the previous RemoteMediator instance, causing the Exception to be thrown
                val remoteKeys = getRemoteKeyForLastItem(state)
                if (remoteKeys == null || remoteKeys.nextKey == null) {
                    throw InvalidObjectException("Remote key should not be null for $loadType")
                }
                remoteKeys.nextKey
            }

        }

        val apiQuery = query + IN_QUALIFIER

        try {
            val apiResponse = service.searchRepos(apiQuery, page, state.config.pageSize)

            val repos = apiResponse.items
            val endOfPaginationReached = repos.isEmpty()
            repoDatabase.withTransaction {
                if (loadType == LoadType.REFRESH) {
                    repoDatabase.remoteKeysDao().clearRemoteKeys()
                    repoDatabase.reposDao().clearRepos()
                }
                val prevKey = if (page == GITHUB_STARTING_PAGE_INDEX) null else page - 1
                val nextKey = if (endOfPaginationReached) null else page + 1
                val keys = repos.map {
                    RemoteKeys(repoId = it.id, prevKey = prevKey, nextKey = nextKey)
                }
                repoDatabase.remoteKeysDao().insertAll(keys)
                repoDatabase.reposDao().insertAll(repos)
            }
            return MediatorResult.Success(endOfPaginationReached = endOfPaginationReached)
        } catch (exception: IOException) {
            return MediatorResult.Error(exception)
        } catch (exception: HttpException) {
            return MediatorResult.Error(exception)
        }
    }

    private suspend fun getRemoteKeyForLastItem(state: PagingState<Int, Repo>): RemoteKeys? {
        return state.pages.lastOrNull() { it.data.isNotEmpty() }?.data?.lastOrNull()
                ?.let { repo ->
                    repoDatabase.remoteKeysDao().remoteKeysRepoId(repo.id)
                }
    }
}

How does InvalidObjectException affect the loading library? I don't think it expects exceptions to occur within its coroutine scope, hence why it exposes MediatorResult.Error (although as append belonging to a generation before the previous refresh, this may as well be expected behavior) — EpicPandaForce, Feb 12 '21 at 17:52
@EpicPandaForce it crashes the app (what we want) or what do you mean? — Florian Walther, Feb 12 '21 at 17:56
If this is standard behavior from Paging 3, then crashing the app seems overkill — EpicPandaForce, Feb 12 '21 at 18:54
@EpicPandaForce it's not standard behavior, it means the programmer messed something up — Florian Walther, Feb 12 '21 at 19:11

score 2 · Answer 1 · answered Feb 13 '21 at 10:24

2

I talked to Dustin Lam and Yigit Boyar and apparently, the best way to handle this is to make prepend an append not depend on the PagingState. This means we should store our remote keys in a table related to a query rather than on the item level.

Exmaple:

@Entity(tableName = "search_query_remote_keys")
data class SearchQueryRemoteKey(
    @PrimaryKey val searchQuery: String,
    val nextPageKey: Int
)

answered Feb 13 '21 at 10:24

Florian Walther

6,237
5
46
104

what if I don't have a search query simple endpoint with page number and limit? – Biro Nader Feb 26 '21 at 19:43
Just store the key in your DB. Or I guess shared preferences or DataStore would also work – Florian Walther Feb 26 '21 at 19:49

Paging 3 - How to handle old APPEND RemoteMediator requests that come trough since beta01?

1 Answers1

Linked