RecyclerView freezes when start to scroll after restoring from background

Question

Here is my adapter class

    public class CategoryAdapter extends RecyclerView.Adapter<CategoryAdapter.ViewHolder> {

    List<CategoryItem> mItems;

    public CategoryAdapter(Context context) {
        super();

        Resources res = context.getResources();
        mItems = new ArrayList<>();

        mItems.add(new CategoryItem(res.getString(R.string.category_book), R.drawable.books));
        // ... few items
        mItems.add(new CategoryItem(res.getString(R.string.category_clothing), R.drawable.clothes));
    }

    @Override
    public ViewHolder onCreateViewHolder(ViewGroup viewGroup, int i) {
        View v = LayoutInflater.from(viewGroup.getContext())
                .inflate(R.layout.item_category_selection, viewGroup, false);
        ViewHolder viewHolder = new ViewHolder(v);
        return viewHolder;
    }

    @Override
    public void onBindViewHolder(ViewHolder viewHolder, int i) {
        CategoryItem category = mItems.get(i);
        viewHolder.title.setText(category.getTitle());
        viewHolder.thumbnail.setImageResource(category.getThumbnail());
    }

    @Override
    public int getItemCount() {
        return mItems.size();
    }

    class ViewHolder extends RecyclerView.ViewHolder {

        public TextView title;
        public ImageView thumbnail;
        public CardView card;

        public ViewHolder(View itemView){
            super(itemView);

            card = (CardView)itemView.findViewById(R.id.card_view);
            title = (TextView)itemView.findViewById(R.id.category_title);
            thumbnail = (ImageView)itemView.findViewById(R.id.category_thumbnail);
        }
    }
}

My fragment

public class CategorySelectionFragment extends Fragment {

    RecyclerView mRecyclerView;
    RecyclerView.LayoutManager mLayoutManager;
    RecyclerView.Adapter mAdapter;

    public CategorySelectionFragment() {  }

    @Override
    public View onCreateView(LayoutInflater inflater, ViewGroup container,
                             Bundle savedInstanceState) {

        View v = inflater.inflate(R.layout.fragment_category_selection, container, false);

        mRecyclerView = (RecyclerView)v.findViewById(R.id.recycler_view);    
        setUpGrid();

        return v;
    }

    private void setUpGrid() {
        mRecyclerView.setHasFixedSize(true);

    colNum = 3;
        mLayoutManager = new GridLayoutManager(getActivity(), colNum);
        mRecyclerView.setLayoutManager(mLayoutManager);

        mAdapter = new CategoryAdapter(getActivity());
        mRecyclerView.setAdapter(mAdapter);

    }

When fragment just created (onCreateView called) recyclerview works perfectly, but when app goes to background and then restore scroll start to freeze on the first few items and then continue to work smoothly.

When app restore from background onCreateView method is not called so I try to set up adapter for recyclerview in onStart(), like this:

  @Override
    public void onStart(){
        super.onStart();

        // Adapter setting here to avoid freezes, when start to scroll after switching to an app from background.
        mRecyclerView.setAdapter(mAdapter);
    } 

This approach has solved the problem, but it seems like not realy clear way to solve that issue. Since for example, restoring the fragment leads to reset last scrolled position.

Thanks for your help!