force ssh to use agent, without fallback to directly accessing the IdentityFile

Question

Company policy requires some ssh keys to be stored securely, e.g. on dedicated USB device. Using keys not stored on the host machine works flawlessly using gnupg with enable-ssh-support, even when multiple keys are used:

Host example.com
    HostName ssh.example.com
    IdentityFile ~/.ssh/smartcard.pub
Host example.net
    HostName git.example.net
    IdentityFile ~/.ssh/another-smartcard.pub
Host example.org
    HostName sftp.example.org
    IdentityFile ~/.ssh/id_rsa.pub

IdentitiesOnly yes
PasswordAuthentication no
PubkeyAuthentication yes

However, when the hardware is unplugged, gpg removes the key from the agent and subsequent ssh calls result in:

Enter passphrase for key '/home/user/.ssh/smartcard.pub':

This seems odd, as both ssh and ssh-agent should be well aware that that file contains a public key only. Is there a good way of making ssh fail verbosely if it has no way of accessing the specified key, instead of asking for a (pointless) passphrase?

Incomplete solutions:

1. remove IdentitiesOnly - ssh will then try all usable keys as expected - but leads to trouble with servers limiting authentication attempts per session
2. wrap ssh in some way alias ssh='grep ^4096 <(ssh-add -l)' && ssh' - works, but will cause headache in case someone ever wants to find out why his ssh setup is broken

Answer 1

Use OpenSSH 7.9 or later, it will at least print some error:

$ ssh example.com
Load key "/home/user/.ssh/smartcard.pub": invalid format
user@example.com: Permission denied (publickey).