While I'm not extremely familiar with the way PIP displays supported actions, you may want to review precisely which actions your MediaSession is reporting.
For some background, I provided a snippet for declaring supported actions in my MediaSession codelab.
In your case, MediaSessionConnector seems to have discovered that there is a playback queue. You may want to instead review how the content of your queue is being provided into your ExoPlayer's media source.
There are two ways of validating that the MediaSession only contains the right actions:
- Media Controller Test is mobile app that allows you to see which actions are reported and test them in a nifty UI.
- You can instead inspect the output of
adb shell dumpsys media_session
:

Note that in this example of YouTube, the playback queue contains 5 items and therefore the action ACTION_SKIP_TO_NEXT is included in the bitwise mask:

For details about these action constants, I can recommend reading more about the actions as declared in PlaybackState.
Once you've confirmed that the MediaSession is reporting the right actions, the PIP display should no longer show skipping next and previous.