Cannot transition entry that is not in the back stack, but currentBackStack.value shows that it is

Question

I'm trying to programmatically go back to the Fragment that called another Fragment. Inside the called Fragment, I used findNavController().popBackStack() to try to got back to the previous Fragment in the backstack, but I'm getting the error I mentioned in the title. Then I called findNavController().currentBackStack.value to see what the backstack looks like, and it looks complete. The called Fragment, the caller Fragment and all the previous Fragment that was shown was in the currentBackStack.value. So, if currentBackStack.value has the complete backstack, then why am I getting an error? I can for sure just add another action for going back to the previous Fragment, but using findNavController().popBackStack() is surely an easier way in the long run

UPDATE: So I created a another action from the called Fragment to the caller Fragment, and it seems that it might not a workaround I could use in my situation. The caller Fragment has an object argument that it receives from another more previous Fragment, and I didn't know that if I made a return action to it, I would need to give the caller Fragment that object again, which the called Fragment doesn't really have a way to easily get access/reference to. Btw, the ActionBar back button works fine, and already hooked to the navigation.

override fun onSupportNavigateUp(): Boolean {
    val navController = findNavController(R.id.nav_host_fragment_content_main)
    return navController.navigateUp(appBarConfiguration) || super.onSupportNavigateUp()
}