Android Paging Library: How to intelligently switch Between online and offline data?

Question

I am following this tutorial by Raywenderlich on paging-library-for-android-with-kotlin on how to use android paging library. This is one of the easiest tutorials on the net and I have followed it thoroughly. However, I would like to make some changes so that I can intelligently switch between online data and offline data.

That is, I have old some posts in my database. Initially I have internet connection. So I load latest data from internet, then insert it into my database. Finally, I show this latest data in my recyclerView / PagedListAdapter. If for some reason, there is no internet connection after sometime, I should show the old posts from database.

How can I do this?

My attempts:

This is my code on github repository.

Here, I tried to create a factory pattern. It checks if initially I have internet, the factory returns pagedList from online dataSource. ELse, the factory returns pagedList from offline dataSource. But this doesnot intelligently switch between the 2 states.

I tried some random codes such as creating a boundary callback. But I am not sure how to make the necessary changes. I am not adding codes here (at least for now) to keep it short and precise.

Can anyone help me?

Edit:

To be specific, I am loading paged data primarily from network. If there is a network error, I don't want to show the user an error. Instead I load paged data from cache / database and continuously show it to my user as long as possible. If the network is back,switch back to network paged data. (that's what instagram / facebook does I think). What is the appropriate way to implement this? See my code / attemp in the answer.

fox example a solution could be the next: If you need to fetch data from your API only when device is connected, always fetch data from API and if you received an UnknownHostException or IOException, then fetch data from your databse — Manuel Mato, Sep 28 '20 at 12:23
yes. the question is how to implement it? it would be nice if there is an example code. — Qazi Fahim Farhan, Sep 28 '20 at 12:45
I saw now your code so your are using retrofit with Call, so you have a success and failure listener methods. Failure receive a throwable param, you need to check the throwable type and if is the appropriate, then fetch data from database. For this, you need to inject your database by constructor. You can see the next approach with retrofit and the flow in the repository (some tips for your approach) https://bitbucket.org/ManuelMato/baseproject/src/develop/app/src/main/java/com/example/manuel/baseproject/ — Manuel Mato, Sep 28 '20 at 13:38
The tutorial you linked looks like it's already doing this. Essentially the DataSource.Factory generated by Room will always load from DB / cached offline data, and BoundaryCallback is triggered to fetch items from network. This means all of Paging is driven by local cached data, which is incrementally updated from network by BoundaryCallback. What issues are you having implementing BoundaryCallback? If you have some specific questions I can try to answer those. — dlam, Sep 29 '20 at 00:21
@dlam yes, you are right, and I understand this tutorial. If I have, say 5 items in db, then room will load those 5, then using boundaryCallBack, it triggers network to load new data. I have implemented it in my practice, no problem there. But I want to load data with network. If the network fails, only then I want to load from db (the exact opposite of the tutorial). This is what I am looking for. — Qazi Fahim Farhan, Sep 29 '20 at 11:47
Are you planning to switch the order in case you hit a network error and want to prioritize loading from db? BoundaryCallback doesn't fetch items to display directly, it stores it in DB and then invalidates to let paging pick up the new items, so it already achieves this without any extra code. — dlam, Sep 29 '20 at 15:37
To be clear, if you're hitting the issue of having stale data in DB, I would simply clear the DB whenever you want to refresh. — dlam, Sep 29 '20 at 15:42

score 0 · Accepted Answer · answered Sep 29 '20 at 12:15

Okay, so after trying out some codes for 2 days, this is what I came up with. However, I really don't know if this is a good pratice or not. So I am open to any acceptable answers.

Explanation:

Since I have multiple data sources(network and database), I created ProfilePostDataSource: PageKeyedDataSource<Pair<Long, Long>, ProfilePost> here the key is a pair, the 1st one for network pagination, the 2nd one is for database pagination.

I used kotlin's Coroutine to write some asynchronous codes in a simple if-else like manner. So we can write it in a psudo-code like this:

Database db;
Retrofit retrofit;

inside loadInitial/loadBefore / loadAfter:
  currNetworkKey = params.key.first;
  currDBKey = params.key.second;
  
  ArrayList<Model> pagedList;

  coroutine{
    ArrayList<Model> onlineList = retrofit.getNetworkData(currNetworkKey);  // <-- we primarily load data from network
    if(onlineList != null) {
      pagedList = onlineList;
      db.insertAll(onlineList);  // <-- update our cache
    }else{
      ArrayList<Model> offlineList = db.getOfflineData(currDBKey); // <-- incase the network fails, we load cache from database  
      if(offlineList !=null){
           pagedList = offlineList;
      }
    }
    if(pagedList != null or empty) {
      nextNetworkKey = // update it accordingly
      nextDBKey = // update it accordingly
      Pair<int, int> nextKey = new Pair(nextNetworkKey, nextDBKey);
      
      pagingLibraryCallBack.onResult(pagedList, nextKey); // <-- submit the data to paging library via callback. this updates your adapter, recyclerview etc...
    }
  }

So in apps like facebook, instagram etc, we see them primarily loading data from network. But if the network is down, they show you a cashed data. We can intelligently make this switch like this code.

Here is a relevant code snippet, the PageKeyedDataSource written in kotlin:

ProfilePostDataSource.kt

/** @brief: <Key, Value> = <Integer, ProfilePost>. The key = pageKey used in api. Value = single item data type in the recyclerView
 *
 * We have a situation. We need a 2nd id to fetch profilePosts from database.
 * Change of plan:  <Key, Value> = < Pair<Int, Int>, ProfilePost>. here the
 *
 *                    key.first = pageKey used in api.      <-- Warning: Dont switch these 2!
 *                     Key.second = db last items id
 *                                   used as out db page key
 *
 * Value = single item data type in the recyclerView
 *
 * */
class ProfilePostDataSource: PageKeyedDataSource<Pair<Long, Long>, ProfilePost> {

  companion object{
    val TAG: String = ProfilePostDataSource::class.java.simpleName;
    val INVALID_KEY: Long = -1;
  }

  private val context: Context;
  private val userId: Int;
  private val liveLoaderState: MutableLiveData<NetworkState>;
  private val profilePostLocalData: ProfilePostLocalDataProvider;

  public constructor(context: Context, userId: Int, profilePostLocalData: ProfilePostLocalDataProvider, liveLoaderState: MutableLiveData<NetworkState>) {
    this.context = context;
    this.userId = userId;
    this.profilePostLocalData = profilePostLocalData;
    this.liveLoaderState = liveLoaderState;
  }

  override fun loadInitial(params: LoadInitialParams<Pair<Long, Long>>, pagingLibraryCallBack: LoadInitialCallback<Pair<Long, Long>, ProfilePost>) {
    val initialNetworkKey: Long = 1L;  // suffix = networkKey cz later we'll add dbKey
    var nextNetworkKey = initialNetworkKey + 1;
    val prevNetworkKey = null; // cz we wont be using it in this case

    val initialDbKey: Long = Long.MAX_VALUE; // dont think I need it
    var nextDBKey: Long = 0L;

    GlobalScope.launch(Dispatchers.IO) {
      val pagedProfilePosts: ArrayList<ProfilePost> = ArrayList(); // cz kotlin emptyList() sometimes gives a weird error. So use arraylist and be happy
      val authorization : String = AuthManager.getInstance(context).authenticationToken;

      try{
        setLoading();
        val res: Response<ProfileServerResponse> = getAPIService().getFeedProfile(
          sessionToken = authorization, id = userId, withProfile = false, withPosts = true, page = initialNetworkKey.toInt()
        );

        if(res.isSuccessful && res.body()!=null) {
          pagedProfilePosts.addAll(res.body()!!.posts);
        }

      }catch (x: Exception) {
        Log.e(TAG, "Exception -> "+x.message);
      }

      if(pagedProfilePosts.isNotEmpty()) {
        // this means network call is successfull
        Log.e(TAG, "key -> "+initialNetworkKey+" size -> "+pagedProfilePosts.size+" "+pagedProfilePosts.toString());

        nextDBKey = pagedProfilePosts.last().id;
        val nextKey: Pair<Long, Long> = Pair(nextNetworkKey, nextDBKey);

        pagingLibraryCallBack.onResult(pagedProfilePosts, prevNetworkKey, nextKey);
        // <-- this is paging library's callback to a pipeline that updates data which inturn updates the recyclerView. There is a line: adapter.submitPost(list) in FeedProfileFragment. this callback is related to that line...
        profilePostLocalData.insertProfilePosts(pagedProfilePosts, userId); // insert the latest data in db
      }else{
        // fetch data from cache
        val cachedList: List<ProfilePost> = profilePostLocalData.getProfilePosts(userId);
        pagedProfilePosts.addAll(cachedList);

        if(pagedProfilePosts.size>0) {
          nextDBKey = cachedList.last().id;
        }else{
          nextDBKey = INVALID_KEY;
        }
        nextNetworkKey = INVALID_KEY; // <-- probably there is a network error / sth like that. So no need to execute further network call. thus pass invalid key
        val nextKey: Pair<Long, Long> = Pair(nextNetworkKey, nextDBKey);
        pagingLibraryCallBack.onResult(pagedProfilePosts, prevNetworkKey, nextKey);

      }
      setLoaded();

    }
  }

  override fun loadBefore(params: LoadParams<Pair<Long, Long>>, pagingLibraryCallBack: LoadCallback<Pair<Long, Long>, ProfilePost>) {}  // we dont need it in feedProflie

  override fun loadAfter(params: LoadParams<Pair<Long, Long>>, pagingLibraryCallBack: LoadCallback<Pair<Long, Long>, ProfilePost>) {
    val currentNetworkKey: Long = params.key.first;
    var nextNetworkKey = currentNetworkKey; // assuming invalid key
    if(nextNetworkKey!= INVALID_KEY) {
      nextNetworkKey = currentNetworkKey + 1;
    }

    val currentDBKey: Long = params.key.second;
    var nextDBKey: Long = 0;

    if(currentDBKey!= INVALID_KEY || currentNetworkKey!= INVALID_KEY) {
      GlobalScope.launch(Dispatchers.IO) {
        val pagedProfilePosts: ArrayList<ProfilePost> = ArrayList(); // cz kotlin emptyList() sometimes gives a weird error. So use arraylist and be happy
        val authorization : String = AuthManager.getInstance(context).authenticationToken;

        try{
          setLoading();
          if(currentNetworkKey!= INVALID_KEY) {
            val res: Response<ProfileServerResponse> = getAPIService().getFeedProfile(
                    sessionToken = authorization, id = userId, withProfile = false, withPosts = true, page = currentNetworkKey.toInt()
            );

            if(res.isSuccessful && res.body()!=null) {
              pagedProfilePosts.addAll(res.body()!!.posts);
            }
          }

        }catch (x: Exception) {
          Log.e(TAG, "Exception -> "+x.message);
        }

        if(pagedProfilePosts.isNotEmpty()) {
          // this means network call is successfull
          Log.e(TAG, "key -> "+currentNetworkKey+" size -> "+pagedProfilePosts.size+" "+pagedProfilePosts.toString());

          nextDBKey = pagedProfilePosts.last().id;
          val nextKey: Pair<Long, Long> = Pair(nextNetworkKey, nextDBKey);

          pagingLibraryCallBack.onResult(pagedProfilePosts,  nextKey);
          setLoaded();
          // <-- this is paging library's callback to a pipeline that updates data which inturn updates the recyclerView. There is a line: adapter.submitPost(list) in FeedProfileFragment. this callback is related to that line...
          profilePostLocalData.insertProfilePosts(pagedProfilePosts, userId); // insert the latest data in db
        }else{
          // fetch data from cache
//          val cachedList: List<ProfilePost> = profilePostLocalData.getProfilePosts(userId);
          val cachedList: List<ProfilePost> = profilePostLocalData.getPagedProfilePosts(userId, nextDBKey, 20);
          pagedProfilePosts.addAll(cachedList);

          if(pagedProfilePosts.size>0) {
            nextDBKey = cachedList.last().id;
          }else{
            nextDBKey = INVALID_KEY;
          }

          nextNetworkKey = INVALID_KEY; // <-- probably there is a network error / sth like that. So no need to execute further network call. thus pass invalid key
          val nextKey: Pair<Long, Long> = Pair(nextNetworkKey, nextDBKey);
          pagingLibraryCallBack.onResult(pagedProfilePosts, nextKey);
          setLoaded();
        }
      }
    }
  }

  private suspend fun setLoading() {
    withContext(Dispatchers.Main) {
      liveLoaderState.value = NetworkState.LOADING;
    }
  }

  private suspend fun setLoaded() {
    withContext(Dispatchers.Main) {
      liveLoaderState.value = NetworkState.LOADED;
    }
  }

}

Thank you for reading this far. If you have a better solution, feel free to let me know. I'm open to any working solutions.

Android Paging Library: How to intelligently switch Between online and offline data?

My attempts:

Edit:

1 Answers1

Explanation:

ProfilePostDataSource.kt