I did some test, and I'm not sure if it's the best solution.
First, I create an azure ad application for using api to access key vault. So I add key-vault-user_impersonation api permission to this app. Then in the key vault access policy tab, I added the application and my user account.

Then with azure ad ropc flow, I got access token for calling key vault api. Finally, I can get secret value with it.

And when I use another account which hasn't added in access policy to get access token, I can't get correct response. And I also tried to add this account to key vault access policy after getting forbidden message, it worked finally. So in my scenario, just both adding azure ad app and specific users in access policy.

By the way, if you consider it not convenient to add many user accounts in key vault access policy, you can add a group with all the users those are needed to access key vault instead. Just creating a group (i.e.named 'permission2xxxkeyvault') in azure ad and set it in access policy tab. I tried it and it worked too.