I had a quick look at this and observed the same (incorrect) behavior; I too was unable to restrict authenticated users to the list of test users:
- Created Apps Script Web app (for a quick win)
- Associated a Cloud Platform project with it
- Enabled Gmail API in the project and added one of its "restricted" scopes
- Added one Google account to test users
I was able to login using the test user and any other Gmail account regardless of whether I included a Gmail restricted scope.
One thing I observed but am unable to explain is that I was not presented with the app's (project's) OAuth Consent Screen. Each time I logged in (incognito), I was prompted by the standard Google login screen only. I expected to be prompted by the OAuth Consent Screen before accessing the app. This likely explains why identities aren't being limited to the test users but I'm unsure why I'm not seeing the consent screen.
Even without Google paid support, you may file issues like this using Google's public issue tracker and these will be seen by Google Engineering.
I recommend you file under "Cloud Platform > Security & Identity" and let someone within Google triage:
https://issuetracker.google.com/issues/new?component=187167&template=1162765