Android Paging 3 Library (3.0.0-alpha03) - How to load ClosestToCurrentPosition items quickly by enabled placeholder with fastscroll bar

Question

I'am using paging 3 library(3.0.0-alpha03), my local cache is Room(160,000 rows).

How to speed up loading "ClosestToCurrentPosition" from local cache items when I use scrollbar and drag to specific position in RecyclerView(GridLayoutManager)?

Suppose I drag scrollbar directly to the position 150,000, the "Flow" map function will be triggered from 0 to 150,000 and that is extreme slow and is not efficient

/**
 * Returns a [PagingData] containing the result of applying the given [transform] to each
 * element, as it is loaded.
 */
@CheckResult
@JvmSynthetic
fun <T : Any, R : Any> PagingData<T>.map(
    transform: suspend (T) -> R
): PagingData<R> = transform { it.map(transform) }

what I expected is something like: when I use scrollbar and drag to 150,000 , the load behavior will load around 154,500~150,500

PagingConfig(pageSize = 500, initialLoadSize = 1000, prefetchDistance = 50, enablePlaceholders = true)

Can someone help me see this issue? or Dustin Lam could u help me? thanks

EDIT: i try to wrap data source as jumping source, but something strange happen. this is my jumping source

class XnMediaFileJumpingPositionPagingSource(

        private val roomPagingSource: PagingSource<Int, SimpleXnMediaFile>

) : PagingSource<Int, SimpleXnMediaFile>() {

    override val jumpingSupported = true

    val roomInvalidateCallback = roomPagingSource.registerInvalidatedCallback {
        Timber.e("[jumping] PagingSource invalidate()")
        invalidate()
        
        // cannot compile
//        unregisterInvalidatedCallback {
//            invalidateCallback
//        }
//        roomPagingSource.unregisterInvalidatedCallback { 
//            roomInvalidateCallback
//        }
    }

    val invalidateCallback = registerInvalidatedCallback {
        Timber.e("[jumping] RoomPagingSource invalidate()")
        roomPagingSource.invalidate()

        // cannot compile
//        unregisterInvalidatedCallback {
//            invalidateCallback
//        }
//        roomPagingSource.unregisterInvalidatedCallback {
//            this.roomInvalidateCallback
//        }
    }

    override suspend fun load(params: LoadParams<Int>): LoadResult<Int, SimpleXnMediaFile> {
        Timber.e("[jumping] Load key %s loadsize %s placeholder %s ", params.key, params.loadSize, params.placeholdersEnabled)
        return roomPagingSource.load(params)
    }

    @ExperimentalPagingApi
    override fun getRefreshKey(state: PagingState<Int, SimpleXnMediaFile>): Int? {
        Timber.e("[jumping] getRefreshKey %s", state.anchorPosition)
        return state.anchorPosition
    }

the params.key keep changing when I drag scroll bar to half of list (maybe is position 80,000 , my db dataset rows amount is 160,000 as I mention above) and finally app ANR

here is the log

2020-07-27 13:06:22.590 12810-12810/com.app.fun E/JumpingPagingSource: [jumping] Load key null loadsize 800 placeholder true 
2020-07-27 13:06:29.107 12810-12810/com.app.fun E/JumpingPagingSource: [jumping] getRefreshKey 85749
2020-07-27 13:06:29.108 12810-12810/com.app.fun E/JumpingPagingSource$invalidateCallback: [jumping] RoomPagingSource invalidate()
2020-07-27 13:06:29.109 12810-12810/com.app.fun E/JumpingPagingSource$roomInvalidateCallback: [jumping] PagingSource invalidate()
2020-07-27 13:06:29.127 12810-12810/com.app.fun E/JumpingPagingSource: [jumping] Load key 85749 loadsize 800 placeholder true 
2020-07-27 13:06:30.920 12810-12810/com.app.fun E/JumpingPagingSource: [jumping] getRefreshKey 171060
2020-07-27 13:06:30.920 12810-12810/com.app.fun E/JumpingPagingSource$invalidateCallback: [jumping] RoomPagingSource invalidate()
2020-07-27 13:06:30.920 12810-12810/com.app.fun E/JumpingPagingSource$roomInvalidateCallback: [jumping] PagingSource invalidate()
2020-07-27 13:06:30.926 12810-12810/com.app.fun E/JumpingPagingSource: [jumping] Load key 171060 loadsize 800 placeholder true 
2020-07-27 13:06:32.808 12810-12810/com.app.fun E/JumpingPagingSource: [jumping] getRefreshKey 85869
2020-07-27 13:06:32.809 12810-12810/com.app.fun E/JumpingPagingSource$invalidateCallback: [jumping] RoomPagingSource invalidate()
2020-07-27 13:06:32.809 12810-12810/com.app.fun E/JumpingPagingSource$roomInvalidateCallback: [jumping] PagingSource invalidate()
2020-07-27 13:06:32.816 12810-12810/com.app.fun E/JumpingPagingSource: [jumping] Load key 85869 loadsize 800 placeholder true 
2020-07-27 13:06:34.572 12810-12810/com.app.fun E/JumpingPagingSource: [jumping] Load key 85200 loadsize 400 placeholder true 
2020-07-27 13:06:34.724 12810-12810/com.app.fun E/JumpingPagingSource: [jumping] getRefreshKey 163199
2020-07-27 13:06:34.725 12810-12810/com.app.fun E/JumpingPagingSource$invalidateCallback: [jumping] RoomPagingSource invalidate()
2020-07-27 13:06:34.725 12810-12810/com.app.fun E/JumpingPagingSource$roomInvalidateCallback: [jumping] PagingSource invalidate()
2020-07-27 13:06:34.729 12810-12810/com.app.fun E/JumpingPagingSource: [jumping] Load key 163199 loadsize 800 placeholder true 
2020-07-27 13:07:06.274 12810-12810/com.app.fun E/JumpingPagingSource: [jumping] Load key 162400 loadsize 400 placeholder true 
2020-07-27 13:07:06.443 12810-12810/com.app.fun E/JumpingPagingSource: [jumping] getRefreshKey 85199
2020-07-27 13:07:06.443 12810-12810/com.app.fun E/JumpingPagingSource$invalidateCallback: [jumping] RoomPagingSource invalidate()
2020-07-27 13:07:06.443 12810-12810/com.app.fun E/JumpingPagingSource$roomInvalidateCallback: [jumping] PagingSource invalidate()
2020-07-27 13:07:06.447 12810-12810/com.app.fun E/JumpingPagingSource: [jumping] Load key 85199 loadsize 800 placeholder true 
2020-07-27 13:07:09.094 12810-12810/com.app.fun E/JumpingPagingSource: [jumping] Load key 84400 loadsize 400 placeholder true 
2020-07-27 13:07:09.236 12810-12810/com.app.fun E/JumpingPagingSource: [jumping] getRefreshKey 162399
2020-07-27 13:07:09.237 12810-12810/com.app.fun E/JumpingPagingSource$invalidateCallback: [jumping] RoomPagingSource invalidate()
2020-07-27 13:07:09.238 12810-12810/com.app.fun E/JumpingPagingSource$roomInvalidateCallback: [jumping] PagingSource invalidate()
2020-07-27 13:07:09.249 12810-12810/com.app.fun E/JumpingPagingSource: [jumping] Load key 162399 loadsize 800 placeholder true 
2020-07-27 13:07:40.931 12810-12810/com.app.fun E/JumpingPagingSource: [jumping] Load key 161600 loadsize 400 placeholder true 
2020-07-27 13:07:41.158 12810-12810/com.app.fun E/JumpingPagingSource: [jumping] getRefreshKey 84419
2020-07-27 13:07:41.158 12810-12810/com.app.fun E/JumpingPagingSource$invalidateCallback: [jumping] RoomPagingSource invalidate()
2020-07-27 13:07:41.159 12810-12810/com.app.fun E/JumpingPagingSource$roomInvalidateCallback: [jumping] PagingSource invalidate()
2020-07-27 13:07:41.162 12810-12810/com.app.fun E/JumpingPagingSource: [jumping] Load key 84419 loadsize 800 placeholder true 
2020-07-27 13:07:43.690 12810-12810/com.app.fun E/JumpingPagingSource: [jumping] Load key 84000 loadsize 400 placeholder true 
2020-07-27 13:07:43.837 12810-12810/com.app.fun E/JumpingPagingSource: [jumping] getRefreshKey 161599
2020-07-27 13:07:43.837 12810-12810/com.app.fun E/JumpingPagingSource$invalidateCallback: [jumping] RoomPagingSource invalidate()
2020-07-27 13:07:43.838 12810-12810/com.app.fun E/JumpingPagingSource$roomInvalidateCallback: [jumping] PagingSource invalidate()
2020-07-27 13:07:43.840 12810-12810/com.app.fun E/JumpingPagingSource: [jumping] Load key 161599 loadsize 800 placeholder true 
2020-07-27 13:08:15.157 12810-12810/com.app.fun E/JumpingPagingSource: [jumping] Load key 160800 loadsize 400 placeholder true 
2020-07-27 13:08:15.338 12810-12810/com.app.fun E/JumpingPagingSource: [jumping] getRefreshKey 84019
2020-07-27 13:08:15.339 12810-12810/com.app.fun E/JumpingPagingSource$invalidateCallback: [jumping] RoomPagingSource invalidate()
2020-07-27 13:08:15.339 12810-12810/com.app.fun E/JumpingPagingSource$roomInvalidateCallback: [jumping] PagingSource invalidate()
2020-07-27 13:08:15.342 12810-12810/com.app.fun E/JumpingPagingSource: [jumping] Load key 84019 loadsize 800 placeholder true 
2020-07-27 13:08:17.980 12810-12810/com.app.fun E/JumpingPagingSource: [jumping] Load key 83600 loadsize 400 placeholder true 
2020-07-27 13:08:18.123 12810-12810/com.app.fun E/JumpingPagingSource: [jumping] getRefreshKey 160799
2020-07-27 13:08:18.124 12810-12810/com.app.fun E/JumpingPagingSource$invalidateCallback: [jumping] RoomPagingSource invalidate()
2020-07-27 13:08:18.125 12810-12810/com.app.fun E/JumpingPagingSource$roomInvalidateCallback: [jumping] PagingSource invalidate()
2020-07-27 13:08:18.136 12810-12810/com.app.fun E/JumpingPagingSource: [jumping] Load key 160799 loadsize 800 placeholder true 
2020-07-27 13:08:49.178 12810-12810/com.app.fun E/JumpingPagingSource: [jumping] Load key 160000 loadsize 400 placeholder true 
2020-07-27 13:08:49.359 12810-12810/com.app.fun E/JumpingPagingSource: [jumping] getRefreshKey 83619
2020-07-27 13:08:49.360 12810-12810/com.app.fun E/JumpingPagingSource$invalidateCallback: [jumping] RoomPagingSource invalidate()
2020-07-27 13:08:49.360 12810-12810/com.app.fun E/JumpingPagingSource$roomInvalidateCallback: [jumping] PagingSource invalidate()
2020-07-27 13:08:49.364 12810-12810/com.app.fun E/JumpingPagingSource: [jumping] Load key 83619 loadsize 800 placeholder true 
2020-07-27 13:08:51.997 12810-12810/com.app.fun E/JumpingPagingSource: [jumping] Load key 83200 loadsize 400 placeholder true 
2020-07-27 13:08:52.148 12810-12810/com.app.fun E/JumpingPagingSource: [jumping] getRefreshKey 159999
2020-07-27 13:08:52.148 12810-12810/com.app.fun E/JumpingPagingSource$invalidateCallback: [jumping] RoomPagingSource invalidate()
2020-07-27 13:08:52.149 12810-12810/com.app.fun E/JumpingPagingSource$roomInvalidateCallback: [jumping] PagingSource invalidate()
2020-07-27 13:08:52.152 12810-12810/com.app.fun E/JumpingPagingSource: [jumping] Load key 159999 loadsize 800 placeholder true 
2020-07-27 13:09:22.903 12810-12810/com.app.fun E/JumpingPagingSource: [jumping] Load key 159200 loadsize 400 placeholder true 
2020-07-27 13:09:23.090 12810-12810/com.app.fun E/JumpingPagingSource: [jumping] getRefreshKey 83219
2020-07-27 13:09:23.090 12810-12810/com.app.fun E/JumpingPagingSource$invalidateCallback: [jumping] RoomPagingSource invalidate()
2020-07-27 13:09:23.090 12810-12810/com.app.fun E/JumpingPagingSource$roomInvalidateCallback: [jumping] PagingSource invalidate()
2020-07-27 13:09:23.094 12810-12810/com.app.fun E/JumpingPagingSource: [jumping] Load key 83219 loadsize 800 placeholder true 
2020-07-27 13:09:25.713 12810-12810/com.app.fun E/JumpingPagingSource: [jumping] Load key 82800 loadsize 400 placeholder true 
2020-07-27 13:09:25.856 12810-12810/com.app.fun E/JumpingPagingSource: [jumping] getRefreshKey 159199
2020-07-27 13:09:25.857 12810-12810/com.app.fun E/JumpingPagingSource$invalidateCallback: [jumping] RoomPagingSource invalidate()
2020-07-27 13:09:25.857 12810-12810/com.app.fun E/JumpingPagingSource$roomInvalidateCallback: [jumping] PagingSource invalidate()
2020-07-27 13:09:25.860 12810-12810/com.app.fun E/JumpingPagingSource: [jumping] Load key 159199 loadsize 800 placeholder true 
2020-07-27 13:09:56.329 12810-12810/com.app.fun E/JumpingPagingSource: [jumping] Load key 158400 loadsize 400 placeholder true 
2020-07-27 13:09:56.494 12810-12810/com.app.fun E/JumpingPagingSource: [jumping] getRefreshKey 82819
2020-07-27 13:09:56.494 12810-12810/com.app.fun E/JumpingPagingSource$invalidateCallback: [jumping] RoomPagingSource invalidate()
2020-07-27 13:09:56.494 12810-12810/com.app.fun E/JumpingPagingSource$roomInvalidateCallback: [jumping] PagingSource invalidate()
2020-07-27 13:09:56.497 12810-12810/com.app.fun E/JumpingPagingSource: [jumping] Load key 82819 loadsize 800 placeholder true 
2020-07-27 13:09:59.132 12810-12810/com.app.fun E/JumpingPagingSource: [jumping] Load key 82400 loadsize 400 placeholder true 
2020-07-27 13:09:59.273 12810-12810/com.app.fun E/JumpingPagingSource: [jumping] getRefreshKey 158399
2020-07-27 13:09:59.273 12810-12810/com.app.fun E/JumpingPagingSource$invalidateCallback: [jumping] RoomPagingSource invalidate()
2020-07-27 13:09:59.273 12810-12810/com.app.fun E/JumpingPagingSource$roomInvalidateCallback: [jumping] PagingSource invalidate()
2020-07-27 13:09:59.276 12810-12810/com.app.fun E/JumpingPagingSource: [jumping] Load key 158399 loadsize 800 placeholder true 
2020-07-27 13:10:29.469 12810-12810/com.app.fun E/JumpingPagingSource: [jumping] Load key 157600 loadsize 400 placeholder true 
2020-07-27 13:10:29.629 12810-12810/com.app.fun E/JumpingPagingSource: [jumping] getRefreshKey 82419
2020-07-27 13:10:29.629 12810-12810/com.app.fun E/JumpingPagingSource$invalidateCallback: [jumping] RoomPagingSource invalidate()
2020-07-27 13:10:29.629 12810-12810/com.app.fun E/JumpingPagingSource$roomInvalidateCallback: [jumping] PagingSource invalidate()
2020-07-27 13:10:29.633 12810-12810/com.app.fun E/JumpingPagingSource: [jumping] Load key 82419 loadsize 800 placeholder true 
2020-07-27 13:10:32.251 12810-12810/com.app.fun E/JumpingPagingSource: [jumping] Load key 82000 loadsize 400 placeholder true 
2020-07-27 13:10:32.397 12810-12810/com.app.fun E/JumpingPagingSource: [jumping] getRefreshKey 157599
2020-07-27 13:10:32.397 12810-12810/com.app.fun E/JumpingPagingSource$invalidateCallback: [jumping] RoomPagingSource invalidate()
2020-07-27 13:10:32.397 12810-12810/com.app.fun E/JumpingPagingSource$roomInvalidateCallback: [jumping] PagingSource invalidate()
2020-07-27 13:10:32.401 12810-12810/com.app.fun E/JumpingPagingSource: [jumping] Load key 157599 loadsize 800 placeholder true 
2020-07-27 13:11:02.096 12810-12810/com.app.fun E/JumpingPagingSource: [jumping] Load key 156800 loadsize 400 placeholder true 
2020-07-27 13:11:02.250 12810-12810/com.app.fun E/JumpingPagingSource: [jumping] getRefreshKey 81999
2020-07-27 13:11:02.250 12810-12810/com.app.fun E/JumpingPagingSource$invalidateCallback: [jumping] RoomPagingSource invalidate()
2020-07-27 13:11:02.250 12810-12810/com.app.fun E/JumpingPagingSource$roomInvalidateCallback: [jumping] PagingSource invalidate()
2020-07-27 13:11:02.253 12810-12810/com.app.fun E/JumpingPagingSource: [jumping] Load key 81999 loadsize 800 placeholder true 
2020-07-27 13:11:04.901 12810-12810/com.app.fun E/JumpingPagingSource: [jumping] Load key 81200 loadsize 400 placeholder true 
2020-07-27 13:11:05.043 12810-12810/com.app.fun E/JumpingPagingSource: [jumping] getRefreshKey 156799
2020-07-27 13:11:05.043 12810-12810/com.app.fun E/JumpingPagingSource$invalidateCallback: [jumping] RoomPagingSource invalidate()
2020-07-27 13:11:05.044 12810-12810/com.app.fun E/JumpingPagingSource$roomInvalidateCallback: [jumping] PagingSource invalidate()
2020-07-27 13:11:05.046 12810-12810/com.app.fun E/JumpingPagingSource: [jumping] Load key 156799 loadsize 800 placeholder true

and PagingConfig

return Pager(
        config = PagingConfig(
                pageSize = 400,
                initialLoadSize = 800,
                prefetchDistance = 160,
                enablePlaceholders = true,
                maxSize = (pageSize + prefetchDistance) * 5,
                jumpThreshold = 4000
        ),
        remoteMediator = XnMediaFileRemoteMediator(xnbay, contract, db),
        pagingSourceFactory = pagingSourceFactory
).flow

Did I miss something ?

dlam · Answer 1 · 2020-07-26T19:35:58.507

2

There is a PagingConfig.jumpThreshold you can set to some value (usually multiple of page size based on how long each page load takes). This will trigger invalidate + REFRESH in case of fast scrolling and pass the user's current scroll position to PagingSource.getRefreshKey(state) via state.anchorPosition.

In order to opt-in to this, you must override PagingSource.jumpingSupported to return true and implement PagingSource.getRefreshKey(state) to return the correct REFRESH load key that will allow the user to restart loading from their current view position.

EDIT: If you're using Room, as of 2.3.0-alpha02, the PagingSource implementation it generates is still using compatibility paths, so you'll need to wrap it with your own implementation to enable jumping.

Something like this:

class JumpingPositionPagingSource(
  val roomPagingSource<Int, Model>
) : PagingSource<Int, Model>() {
    val roomInvalidateCallback = roomPagingSource.registerInvalidatedCallback {
      invalidate()
      unregisterInvalidatedCallback(invalidateCallback)
      roomPagingSource.unregisterInvalidatedCallback(roomInvalidateCallback)
    }
    val invalidateCallback = registerInvalidatedCallback {
      roomPagingSource.invalidate()
      unregisterInvalidatedCallback(invalidateCallback)
      roomPagingSource.unregisterInvalidatedCallback(roomInvalidateCallback)
    }

  override suspend fun load(loadParams: LoadParams<Int>) {
    return roomPagingSource.load(loadParams)
  }

  override val jumpingSupported = true

  override fun getRefreshKey(state: PagingState<Int, Model>): Int? {
    return state?.anchorPosition
  }
}

edited Jul 26 '20 at 19:35

answered Jul 25 '20 at 19:30

dlam

3,547
17
20

Hi, do u mean I need to custom a local CustomDBSource that extend PagingSource and manual load data via DAO in load() func ? Do u have an example about custom db source ? thanks – kevinstyle Jul 26 '20 at 04:10
How to set jumpingSupported = true if I use Room as local cache ? java.lang.IllegalArgumentException: PagingConfig.jumpThreshold was set, but the associated PagingSource has not marked support for jumps by overriding PagingSource.jumpingSupported to true. – kevinstyle Jul 26 '20 at 13:24
The easiest way is probably to create a wrapping PagingSource, the implementation generated by Room is still using compatibility paths, but we have plans to fix in the future. I'll edit my answer with an untested sample to get you started. – dlam Jul 26 '20 at 19:32
I try the wrapping source and something strange happen, I edit the question above. please do me a favor to see what's the problem. thanks – kevinstyle Jul 27 '20 at 05:20
Are you able to upload your project or share it with me somehow? If you want to keep it private you can upload it to github, my username is dlam there. – dlam Jul 27 '20 at 16:31
I've invited you as collaborator, please check github notification, thanks, but kindly remind this is private project, thanks. – kevinstyle Jul 28 '20 at 07:46
1

To follow up here, next version of paging will enable jumping on paging source from Room. – dlam Jul 29 '20 at 20:35
great, glad to hear the news, i'll continue to follow this update. and did u see my project's code? Is it the wrong way to use or is this feature not yet worked properly? – kevinstyle Jul 30 '20 at 05:07
Hi, when will be the next version released? I need the jumping paging source for Room. thanks. – kevinstyle Aug 03 '20 at 12:45
We typically release biweekly, but aren't allowed to promise any dates as we sometimes delay for bigger releases. Sorry I haven't gotten a chance to reproduce using your sample yet - your project is quite large! – dlam Aug 04 '20 at 17:03
This implementation of `getRefreshKey` can't be right. `state.anchorPosition` is an index in the list; it is always an `Int?`. The return value of `getRefreshKey` is supposed to be a key for loading a specific page. If you change the key type for your paging source class to something other than `Int`, this code won't even compile. – Ted Hopp Nov 11 '20 at 13:25

Android Paging 3 Library (3.0.0-alpha03) - How to load ClosestToCurrentPosition items quickly by enabled placeholder with fastscroll bar

1 Answers1