2

I have an application where i implemented paging library 3 to fetch data from api and paginate it , it works fine fetching data , the next implementation was to store the fetched data in room database , i have created the remotemediator class and wrote the code to store data , but the issue is that it stores only values of first page ( for example in my case im using the movie db api , each page fetched has 20 movies , and there are many pages ) , in my case it only saves the first 20 movies , even when i scroll , it is not storing more data , i have implemented the same exact code but seems to be the case , i faced it in an older project and now this one , i need some help , thank you in advance.

  • Movies Dao
 @Dao
interface MoviesDao {

    @Query("SELECT * FROM movieTable ORDER BY id")
     fun getMovies() : PagingSource<Int,Result>


    @Insert(onConflict = OnConflictStrategy.REPLACE)
    suspend fun insertMovies(result: List<Result>)


    @Query("DELETE FROM movieTable")
    suspend fun clearMovies()


}
  • RemoteKeys Dao
@Dao
interface RemoteKeysDao {

    @Insert(onConflict = OnConflictStrategy.REPLACE)
    suspend fun insertAll(remoteKey: List<RemoteKeys>)

    @Query("SELECT * FROM remote_keys WHERE movieId = :movieId")
    suspend fun remoteKeysRepoId(movieId : Long): RemoteKeys?

    @Query("DELETE FROM remote_keys")
    suspend fun clearRemoteKeys()


}
  • RemoteMediator Class
private  var MOVIES_API_STARTING_PAGE_INDEX = 1
@ExperimentalPagingApi
class MoviesMediator(
    private var authResponse: AuthResponse,
    private  var movieDatabase: MovieDatabase
) : RemoteMediator<Int,Result>() {

    override suspend fun load(loadType: LoadType, state: PagingState<Int, Result>): MediatorResult {
        val page = when (loadType) {
            LoadType.REFRESH -> {
                val remoteKeys = getRemoteKeyClosestToCurrentPosition(state)
                remoteKeys?.nextKey?.minus(1) ?: MOVIES_API_STARTING_PAGE_INDEX
            }
            LoadType.PREPEND -> {
                val remoteKeys = getRemoteKeyForFirstItem(state)
                val prevKey = remoteKeys?.prevKey
                if (prevKey == null) {
                    return MediatorResult.Success(endOfPaginationReached = remoteKeys != null)
                }
                prevKey
            }
            LoadType.APPEND -> {
                val remoteKeys = getRemoteKeyForLastItem(state)
                val nextKey = remoteKeys?.nextKey
                if (nextKey == null) {
                    return MediatorResult.Success(endOfPaginationReached = remoteKeys != null)
                }
                nextKey
            }
        }
        try {
            val response = authResponse.getMovies(Constants.API_KEY, Constants.LANGUAGE, page).results


            val endOfPagination = response.isEmpty()
            movieDatabase.withTransaction {
                // clear all tables in the database
                if (loadType == LoadType.REFRESH) {
                    movieDatabase.remoteKeysDao().clearRemoteKeys()
                    movieDatabase.MovieDao().clearMovies()
                }
                val prevKey = if (page == MOVIES_API_STARTING_PAGE_INDEX) null else page - 1
                val nextKey = if (endOfPagination) null else page + 1

                val keys = response.map {
                    RemoteKeys(movieId = it.movieID, prevKey = prevKey, nextKey = nextKey)
                }
                movieDatabase.remoteKeysDao().insertAll(keys)
                movieDatabase.MovieDao().insertMovies(response)
            }
            return MediatorResult.Success(endOfPaginationReached = endOfPagination)
        } catch (ex: Exception) {
            return MediatorResult.Error(ex)
        }
    }

    private suspend fun getRemoteKeyForFirstItem(state: PagingState<Int, Result>): RemoteKeys? {
        // Get the last page that was retrieved, that contained items.
        // From that last page, get the last item
        return state.pages.firstOrNull() { it.data.isNotEmpty() }?.data?.firstOrNull()
            ?.let { movieId ->
                // Get the remote keys of the last item retrieved
                movieDatabase.remoteKeysDao().remoteKeysRepoId(movieId.movieID)
            }
    }
    private suspend fun getRemoteKeyClosestToCurrentPosition(state: PagingState<Int, Result>): RemoteKeys? {
        // The paging library is trying to load data after the anchor position
        // Get the item closest to the anchor position
        return state.anchorPosition?.let { position ->
            state.closestItemToPosition(position)?.movieID?.let { movieId ->
                movieDatabase.remoteKeysDao().remoteKeysRepoId(movieId = movieId)
            }
        }
    }

    private suspend fun getRemoteKeyForLastItem(state: PagingState<Int, Result>): RemoteKeys? {
        // Get the last page that was retrieved, that contained items.
        // From that last page, get the last item
        return state.pages.lastOrNull() { it.data.isNotEmpty() }?.data?.lastOrNull()
                ?.let { repo ->
                    // Get the remote keys of the last item retrieved
                    movieDatabase.remoteKeysDao().remoteKeysRepoId(movieId = repo.movieID)
                }
    }

}
  • Passing RemoteMediator to paging data
    val dataFlow : kotlinx.coroutines.flow.Flow<PagingData<Result>> =
        Pager(getPagingConfig(),
        remoteMediator = MoviesMediator(authResponse,movieDatabase)){
            MoviePagingSource(authResponse)
        }.flow
            .cachedIn(viewModelScope)

  • Showing data in MainActivity

 @ExperimentalPagingApi
    private fun setUpAdapterOnline(){
        moviesAdapter = MoviesAdapter()
        lifecycleScope.launchWhenStarted {
            moviesModel.dataFlow.collectLatest {
                moviesAdapter.submitData(it)
            }
        }

        binding.recycler.adapter = moviesAdapter
        binding.recycler.adapter =  moviesAdapter.withLoadStateHeaderAndFooter(
            header = LoadingStateAdapter { moviesAdapter.retry() },
            footer = LoadingStateAdapter { moviesAdapter.retry() }
        )
    }
Taki
  • 3,290
  • 1
  • 16
  • 41
  • Could you also share how you use the `Flow>` ? Are you using `collectLatest` to observe it? Also, are you getting calls to `RemoteMediator` for `APPEND` / `PREPEND` at all? – dlam May 17 '21 at 19:27
  • Yeah i'm using collectLatest , for append and prepend , i think they are only called once , i'm not much familiar with paging library 3 but i have put a log where i push data into room in the append part , called first time only ( i mean when first 20 movies are loade ) – Taki May 18 '21 at 05:48
  • I noticed you have two Pagers, one for offline and one for online which seems incorrect to me. Everything in Paging is driven by PagingSource, so you don't need both. RemoteMediator is basically a callback - if you want to use offline data, you can simply attempt the network fetch on remote refresh and only clear + insert if it succeeds. – dlam May 19 '21 at 00:42
  • If you can share how you are mixing multiple / using the `Flow` here, I can try to help more, but it's not really enough information as it is. – dlam May 19 '21 at 00:43
  • Actually I also just noticed in your "online" version with RemoteMediator, your `pagingSourceFactory` is also different. What does `MoviePagingSource()` look like? You should be using the one provided from Room since you are inserting into Room and using that to drive Paging. – dlam May 19 '21 at 00:45
  • i got it now , so i have to use just the same call that fetchs data and also within that call i added the remotemediator callback , but i m not sure if i ve seen refresh remote code in the codelabs , do you please have any template or code on how to do that , because you re right , im fetching data from netwok and database in different , and you said they shoul be in the same code block – Taki May 20 '21 at 20:59
  • actually MoviePagingSource() is the main call which gets data from api call , and the offlineDataFlow() is function where i get the save data from room but as you said that's not correct and i should implement remote refresh which i don't really know how , any example code to implement that , thank you . – Taki May 20 '21 at 21:09
  • Your `RemoteMediator` should do the network fetches and update DB via Room. Then you should use PagingSource from Room to drive Paging (which automatically observes for data changes made by RemoteMediator and invalidates to notify paging of updates). Both our codelab (https://developer.android.com/codelabs/android-paging#14) and our github sample (https://github.com/android/architecture-components-samples/blob/main/PagingWithNetworkSample/app/src/main/java/com/android/example/paging/pagingwithnetwork/reddit/repository/inDb/PageKeyedRemoteMediator.kt) has some good examples of RemoteMediator – dlam May 24 '21 at 22:29
  • I have removed the offline function wich retreives the data from db and i did as you said , i m only using one function which combines both , api call + remoteMediator callback , i have changed some code , can you please check it out again and tell me how is it possible to call the saved data from room when there is no internet ( trigger data from local cache ) thank you – Taki May 25 '21 at 17:31
  • i found out that page ( from load type is not incrementing at all ) , that's why only first page is stored , any reason why please , im checking th codelabs over an over but coulnt figure out – Taki May 25 '21 at 17:55

1 Answers1

0

UPDATE

I tried using this NewsApi instead of this Unsplash Api. Even though the NewsApi provides the per_page as a query parameter, still I faced similar problem where only first 2 pages load and after that on debugging it shows that the APPEND block inside the Load method is getting called over and over and the value assigned to page is always 2. Hence the next pages aren't getting fetched at all.

ORIGINAL ANSWER

@dlam is right, you need to follow the Single source of truth principle here, and only request data from the Paging Source inside MoviesDao. I did not find the exact solution for your problem, but I do have some findings to share.

I would say that I debugged the Remote Mediator in the sample codelab here and I found that it actually increments the page as you scroll. However, when debugging your code from the github link you sent me, the value of page never increments and the APPEND block inside MoviesMediator is executed over and over, even when not scrolling. I tried to find the reason and the only thing I can come up with is the Api that you are using. I tried the same feature, saving from network to database but using the Unsplash Api and I was able to view all the pages when offline too (not just one page unlike your case).

I'm not sure if it is the per_page parameter that the codelab also passed as an argument inside the Network/Api Service Interface. In my case, the Unsplash Api also allowed to pass per_page as an argument.

Here I am attaching a link to my github repository. Try comparing the results and feel free to ask any more doubts.

mehul bisht
  • 682
  • 5
  • 15
  • there could be an issue with the api yeah but with paging library 2 i could save multiple pages , so i'm not really sure why – Taki May 25 '21 at 23:33
  • The project I linked is able to save multiple pages as well with Paging 3. Take a look :) – mehul bisht May 25 '21 at 23:47