Android Keyboard Accessibility: focus is going to entire top toolbar instead of its child

Question

I am working on an Android app that has a keyboard accessibility feature to access the app using the keyboard.

I am using

androidx.appcompat.widget.Toolbar

It has one navigation button.

val navigationButton = topToolbar.getChildAt(0) as ImageButton
navigationButton.importantForAccessibility = IMPORTANT_FOR_ACCESSIBILITY_YES
navigationButton.focusable = View.FOCUSABLE
navigationButton.isClickable = true

However, when I try to focus on the navigation button of my App toolbar, it is getting focused on the whole toolbar instead of that cross button. After pressing the 'TAB' key on the keyboard, then the focus is coming to the cross button.

I don't want the whole toolbar to get focused. I only want the cross button to get focussed.

Here is my toolbar:

<androidx.appcompat.widget.Toolbar
    android:id="@+id/my_fragment_top_toolbar"
    android:layout_width="match_parent"
    android:layout_height="wrap_content"
    android:elevation="4dp"
    app:navigationIcon="@drawable/close_icon"
    app:title="" />