3

I am struggling with the Paging 3 Library of Jetpack.

I setup

  • Retrofit for the network API calls
  • Room to store the retrieved data
  • A repository that exposes the Pager.flow (see code below)
  • A RemoteMediator to cache the network results in the room database

The PagingSource is created by Room.

I understand that the RemoteMediators responsibility is to fetch items from the network and persist them into the Room database. By doing so, we can use the Room database as single point of truth. Room can easily create the PagingSource for me as long as I am using Integers as nextPageKeys.

So far so good. Here is my ViewModel to retrieve a list of Sources:

    private lateinit var _sources: Flow<PagingData<Source>>
    val sources: Flow<PagingData<Source>>
        get() = _sources

    
    private fun fetchSources() = viewModelScope.launch {
        _sources = sourcesRepository.getSources(
            selectedRepositoryUuid,
            selectedRef,
            selectedPath
        )
    }

val sources is collected in the Fragment.

fetchSources() is called whenever one of the three parameters change (selectedRepositoryUuid, selectedRef or selectedPath)

Here is the Repository for the Paging call

    fun getSources(repositoryUuid: String, refHash: String, path: String): Flow<PagingData<Source>> {
        return Pager(
            config = PagingConfig(50),
            remoteMediator = SourcesRemoteMediator(repositoryUuid, refHash, path),
            pagingSourceFactory = { sourcesDao.get(repositoryUuid, refHash, path) }
        ).flow
    }

Now what I experience is that Repository.getSources is first called with correct parameters, the RemoteMediator and the PagingSource are created and all is good. But as soon as one of the 3 parameters change (let's say path), neither the RemoteMediator is recreated nor the PagingSource. All requests still try to fetch the original entries.

My question: How can I use the Paging 3 library here in cases where the paging content is dependent on dynamic variables?

If it helps to grasp my use-case: The RecyclerView is displaying a paged list of files and folders. As soon as the user clicks on a folder, the content of the RecyclerView should change to display the files of the clicked folder.


Update:

Thanks to the answer of dlam, the code now looks like this. The code is a simplification of the real code. I basically encapsulate all needed information in the SourceDescription class.:

ViewModel:

    private val sourceDescription = MutableStateFlow(SourceDescription())

    fun getSources() = sourceDescription.flatMapConcat { sourceDescription ->

        // This is called only once. I expected this to be called whenever `sourceDescription` emits a new value...?

        val project = sourceDescription.project
        val path = sourceDescription.path

        Pager(
            config = PagingConfig(30),
            remoteMediator = SourcesRemoteMediator(project, path),
            pagingSourceFactory = { sourcesDao.get(project, path) }
        ).flow.cachedIn(viewModelScope)
    }

    fun setProject(project: String) {
        viewModelScope.launch {
            val defaultPath = Database.getDefaultPath(project)
            val newSourceDescription = SourceDescription(project, defaultPath)
            sourceDescription.emit(newSourceDescription)
        }
    }

In the UI, the User first selects a project, which is coming from the ProjectViewModel via LiveData. As soon as we have the project information, we set it in the SourcesViewModel using the setProject method from above.

Fragment:

    override fun onViewCreated(view: View, savedInstanceState: Bundle?) {
        super.onViewCreated(view, savedInstanceState)

        // load the list of sources
        viewLifecycleOwner.lifecycleScope.launchWhenStarted {
            sourcesViewModel.getSources().collectLatest { list ->
                sourcesAdapter.submitData(list) // this is called only once in the beginning
            }
        }

        projectsViewModel.projects.observe(viewLifecycleOwner, Observer { project -> 
            sourcesViewModel.setProject(project)
        })
    }
muetzenflo
  • 5,653
  • 4
  • 41
  • 82

1 Answers1

10

The overall output of Paging is a Flow<PagingData>, so typically mixing your signal (file path) into the flow via some flow-operation will work best. If you're able to model the path the user clicks on as a Flow<String>, something like this might work:

ViewModel.kt

class MyViewModel extends .. {
  val pathFlow = MutableStateFlow<String>("/")
  val pagingDataFlow = pathFlow.flatMapLatest { path ->
    Pager(
      remoteMediator = MyRemoteMediator(path)
      ...
    ).flow.cachedIn(..)
  }
}

RemoteMediator.kt

class MyRemoteMediator extends RemoteMediator<..> {
  override suspend fun load(..): .. {
    // If path changed or simply on whenever loadType == REFRESH, clear db.
  }
}

The other strategy if you have everything loaded is to pass the path directly into PagingSource, but it sounds like your data is coming from network so RemoteMediator approach is probably best here.

dlam
  • 3,547
  • 17
  • 20
  • Thank you very much for this input! I am a step further now. But I have two follow-up questions, because it does not work yet: 1. The compiler only allows the use of `flatMapConcat` instead of `flatMap`. Does that make a significant difference? 2. If I call within my ViewModel `pathFlow.emit("/new/path")` then `pagingDataFlow` is not updated, although I am collecting it in my Fragment. Do you have an idea why? I will update my question to give you my current code. – muetzenflo Mar 14 '21 at 20:39
  • Ah sorry I forgot that `Flow.flatMap` is deprecated now, `flatMapConcat` is the right one as `flatMapMerge` is for concurrenct limit > 1. – dlam Mar 14 '21 at 20:50
  • For second issue - are you collectin from pagingDataFlow using `collect` or `collectLatest`? Make sure to use `collectLatest` as it will cancel previous generation once new one arrives since `.submitData()` may run for some time before invalidation closes it. – dlam Mar 14 '21 at 20:52
  • I tried both `collect` and `collectLatest`. None worked. I updated the code in my question. Highly appreciate your comments! – muetzenflo Mar 14 '21 at 20:53
  • If that's not your issue I need a bit more information... maybe some test code on `pagingDataFlow`'s emissions that shows what your expected behavior to be? – dlam Mar 14 '21 at 20:53
  • Please make sure your Fragment code is up to date too as it still shows usage of `collect`. – dlam Mar 14 '21 at 20:55
  • Oops sorry, you should also use `flatMapLatest` instead of `flatMap`. My bad there. – dlam Mar 14 '21 at 20:56
  • O_O something happens!! Still not working end2end, but now it is more of an API issue. I can't believe it haha. Was hitting my head against this for days now. I hope to finish this in the next days and if the Flow now works will gladly spend some bounty here for your effort and quick reactions! Thanks a lot, I love this community :) – muetzenflo Mar 14 '21 at 21:03
  • No problem, glad it works for you now! :) – dlam Mar 14 '21 at 21:16
  • 1
    There is a problem with this solution: if the activity is recreated or if the user leaves the app and goes back to it, the flow collection will restart. MutableStateFlow will then re-emit its latest value, causing a new Pager instance being created. So caching the paging data is useless because everything will be loaded from scratch on each new Flow collection. – BladeCoder May 16 '21 at 14:19
  • @BladeCoder If your app is completely destroyed / gc'd there is nothing you can do to preserve in-memory cache - it has to happen on disk via save / restore instance state. At this point however, you might as well just load from disk directly via `PagingSource`. My answer above is targeting caching in-memory through `ViewModel`, which can survive past fragment transitions or config changes (often has a "greater" lifecycle than directly in UI) – dlam Aug 30 '21 at 20:00
  • @dlam I'm not talking about the app being destroyed or gc'ed. I'm talking about a problem inherent to using StateFlow in ViewModels: each new collection of a Flow exposed in a ViewModel will result in the underlying StateFlow trigger emitting a new value, effectively discarding the cache which was still valid. I wrote a long blog post describing the issue: https://bladecoder.medium.com/kotlins-flow-in-viewmodels-it-s-complicated-556b472e281a – BladeCoder Aug 31 '21 at 13:50
  • @BladeCoder Ah you're talking about `val pathFlow = MutableStateFlow("/")`. Sorry I completely misread your comments earlier. You're right that any collection would cause StateFlow to replay and unnecessarily create a new Pager deleting the cache. I should update my sample :) – dlam Sep 09 '21 at 02:17