Resource efficient combination of RecyclerView and LiveDate for datasets with large objects

Question

Main question: What is the correct approach to use an Android RecyclerView and RecyclerView.Adapter without holding the whole dataset in memory?

Background

Before I explain my initial question in more detail, let me first elaborate on the background which will help to understand my question more easily. It seems that there are basically two approaches how to deal with RecyclerView.Adapter and LiveData from a ViewModel:

1. The "hackish" or "quick-and-dirty" way

   It it is not possible to use a RecyclerView.Adapter or even a single ViewHolder as a LiveData observer out-of-the-box, because neither RecyclerView.Adapter nor ViewHolder are a LifecycleOwner. The internet is full of workarounds, how a ViewHolder can observe its own data either by making a ViewHolder a LifecycleOwner like in this example or by observing the LifeData forever like in this question. "Observing forever" does not require a LifecycleOwner, but of course this raises the problem, how a ViewHolder stops observing again.

2. The "official" way

   According to this answer one should not use LiveData with a RecyclerView.Adapter directly. Instead, the activity/fragment which holds the RecyclerView should observe a LifeData of the whole dataset and notify the RecyclerView.Adapter, if the dataset has changed. In order to keep the update small for the RecyclerView a 3-way diff between the old and the new dataset can be implemented. Here's an example of observing LiveData of a whole dataset in the fragment and using DiffUtil within a RecyclerView adapter.

Detailed question

The official way has two obvious drawbacks and the second one actually makes it impossible for me to use it.

1. Typically, the underlying repository is able to directly return the total number of entries and to return the i'th object. There is no need to fetch the whole dataset first and then use a copy of the the dataset just to pick out the currently visible objects. Moreover, the repository typically "knows" which item has been inserted, removed or updated. Hence, it feels wrong to let the Fragment observe the complete dataset, let the ViewModel post a new version of the complete dataset to the fragment (by MutableLiveData.setValue( List<Entity> )) and then let the fragment run a 3-way diff between the old and the new dataset in order to find out, what has changed, if the repository already knows these information.

2. The great benefit of a RecyclerView is that is does not create unnecessary ViewHolder objects for each entry, but only as many ViewHolder objects as are visible at the same time and then "recycles" those objects to show new data if the user scrolls through the list. However, this is not true for the underlying dataset, if the the official approach is used. The fragment observes the whole dataset and thus each entity of the dataset must be kept in memory. This is prohibitive, if the the dataset is large or if each entity is a complex object and not only a bunch of POD-types. In that case, it would be really beneficial if the "recycle approach" would stretch out over the dataset, too.

In my particular use case, each entity holds a Drawable. A ViewHolder basically contains a ImageView which shall display that drawable, when the ViewHolder comes into view. In other words, my onBindViewHolder(ViewHolder holder, int position) basically would look like that, if I were following the official approach

holder.getImageView().setImageDrawable( mDataset.get( position ).getDrawable() )

Here mDataset is a deep copy of the whole dataset. Moreover, the content of each drawable depends on the ID of the entity and is created programmatically. Hence, there is no real reason to keep all these drawables in an in-memory dataset. The following approach would be more efficient:

1. On binding, the ViewHolder obtains a LifeData of the Drawable from the ViewModel.
2. The ViewHolder places the Drawable in the ImageView
3. The ViewHolder starts observing the Drawable.
4. Under the hood, when the ViewModel provides a LifeData of the Drawable to the ViewHolder, the ViewModel triggers the repository to render the real drawable.
5. The repository renders the final Drawable in a background thread.
6. When the thread has rendered the final Drawable, the ViewModel updates the LiveData
7. The ViewHolder receives a notification and displays the final content in its ImageView.

In other words, a fake code would look like that

class ViewHolder ... {
   ...
   public void onBindViewHolder( MyViewHolder holder, int position ) {
      // N.b. requesting i'th entity returns a placeholder entity
      // Such an entity has a drawble of correct size, but the drawable
      // shows an illustration which indicates that is has not been finally rendered
      // At the same time, `viewModel.getEntity(i)` starts the rendering process
      // in the background
      LiveData<Entity> liveEntity = viewModel.getEntity( position );
      holder.getImageView().setImageDrawable( liveEntity.getValue.getDrawable() );
      liveEntity.observe( holder, holder );
   }
}

class MyViewHolder ... implements Observer<Entity> {
  ...
  public void onChanged( Entity entity ) {
    // Method is called, if background thread has rendered the final drawable
    getImageView().setImageDrawable( entity.getDrawable )
  }
}

Can this be achieved?