I have been trying to overcome this kind of issue in a few apps as well. My research points me here: Android - Activity Lifecycle as it states this:
However, if the system destroys the activity due to system constraints (rather than normal app behavior), then although the actual Activity instance is gone, the system remembers that it existed such that if the user navigates back to it, the system creates a new instance of the activity using a set of saved data that describes the state of the activity when it was destroyed. The saved data that the system uses to restore the previous state is called the instance state and is a collection of key-value pairs stored in a Bundle object.
So, we need to be leveraging the savedInstanceState Bundle for cleanly packing and unpacking each Activity and Fragment.
I'll work on an example for you ASAP.