SSH agent forwarding fails depending on intermediaries

Question

I am getting a permission denied when trying to connect over SSH with agent forwarding. I have four machines: my local box (me), a private server (private), a public gateway that can route to the private server (gateway), and a public server (destination).

If I need to connect to "destination" or "gateway", I can simply do the following, which verifies my key is valid for both hosts:

me@me:~$ ssh -i my.key destination
me@me:~$ ssh -i my.key gateway

If I need to connect to "private", I can do the following, which verifies agent forwarding is working as expected and my key is valid for the host:

me@me:~$ ssh -Ati my.key gateway
me@gateway:~$ ssh private

If I need to connect to "destination" from "gateway", I can do the following, which verifies agent forwarding works when connecting to destination:

me@me:~$ ssh -Ati my.key gateway
me@gateway:~$ ssh destination

If I need to connect to "destination" from "private", I should be able to do the following:

me@me:~$ ssh -Ati my.key gateway
me@gateway:~$ ssh -At private
me@private:~$ ssh destination

But when I try that last command, I receive Permission denied (publickey). If I run with verbosity turned up, I get the following:

OpenSSH_6.6.1, OpenSSL 1.0.1k-fips 8 Jan 2015
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 56: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to destination [52.35.125.100] port 22.
debug1: Connection established.
debug1: identity file /home/me/.ssh/id_rsa type -1
debug1: identity file /home/me/.ssh/id_rsa-cert type -1
debug1: identity file /home/me/.ssh/id_dsa type -1
debug1: identity file /home/me/.ssh/id_dsa-cert type -1
debug1: identity file /home/me/.ssh/id_ecdsa type -1
debug1: identity file /home/me/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/me/.ssh/id_ed25519 type -1
debug1: identity file /home/me/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.6.1
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1
debug1: match: OpenSSH_6.6.1 pat OpenSSH_6.6.1* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug3: load_hostkeys: loading entries for host "destination" from file "/home/me/.ssh/known_hosts"
debug3: load_hostkeys: found key type ECDSA in file /home/me/.ssh/known_hosts:7
debug3: load_hostkeys: loaded 1 keys
debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-dss-cert-v00@openssh.com,ssh-ed25519,ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: first_kex_follows 0 
debug2: kex_parse_kexinit: reserved 0 
debug2: kex_parse_kexinit: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: first_kex_follows 0 
debug2: kex_parse_kexinit: reserved 0 
debug2: mac_setup: setup hmac-md5-etm@openssh.com
debug1: kex: server->client aes128-ctr hmac-md5-etm@openssh.com none
debug2: mac_setup: setup hmac-md5-etm@openssh.com
debug1: kex: client->server aes128-ctr hmac-md5-etm@openssh.com none
debug1: kex: curve25519-sha256@libssh.org need=16 dh_need=16
debug1: kex: curve25519-sha256@libssh.org need=16 dh_need=16
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ECDSA 01:2f:4e:3f:b3:a3:92:0f:29:e0:72:d7:d1:2c:af:4b
debug3: load_hostkeys: loading entries for host "destination" from file "/home/me/.ssh/known_hosts"
debug3: load_hostkeys: found key type ECDSA in file /home/me/.ssh/known_hosts:7
debug3: load_hostkeys: loaded 1 keys
debug3: load_hostkeys: loading entries for host "52.35.125.100" from file "/home/me/.ssh/known_hosts"
debug3: load_hostkeys: found key type ECDSA in file /home/me/.ssh/known_hosts:8
debug3: load_hostkeys: loaded 1 keys
debug1: Host 'destination' is known and matches the ECDSA host key.
debug1: Found key in /home/me/.ssh/known_hosts:7
debug1: ssh_ecdsa_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: me@me (0x5611e148efd0),
debug2: key: /home/me/.ssh/id_rsa ((nil)),
debug2: key: /home/me/.ssh/id_dsa ((nil)),
debug2: key: /home/me/.ssh/id_ecdsa ((nil)),
debug2: key: /home/me/.ssh/id_ed25519 ((nil)),
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: me@me
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey
debug1: Trying private key: /home/me/.ssh/id_rsa
debug3: no such identity: /home/me/.ssh/id_rsa: No such file or directory
debug1: Trying private key: /home/me/.ssh/id_dsa
debug3: no such identity: /home/me/.ssh/id_dsa: No such file or directory
debug1: Trying private key: /home/me/.ssh/id_ecdsa
debug3: no such identity: /home/me/.ssh/id_ecdsa: No such file or directory
debug1: Trying private key: /home/me/.ssh/id_ed25519
debug3: no such identity: /home/me/.ssh/id_ed25519: No such file or directory
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
Permission denied (publickey).

Note: I've attempted to sanitize the hosts, users, addresses etc. before posting; I may have made a transcription error.

I've also verified SSH_AUTH_SOCK looks properly exported on the intermediary systems:

me@gateway:~$ echo $SSH_AUTH_SOCK     # /tmp/ssh-X5p4HjmGhP/agent.918
me@private:~$ echo $SSH_AUTH_SOCK     # /tmp/ssh-aR5WfnzmgE/agent.14429

What should I be looking for to identify what's wrong?

BTW, you could use `ProxyCommand so that your client makes a direct connection to the ultimate destination. http://serverfault.com/a/72347/984 — Zoredache, Oct 04 '16 at 19:18
I typically use ProxyCommand to connect to "private" through "gateway", but I didn't want that to confuse the issue. The primary reason I use agent forwarding is so I can avoid pre-configuring connections; otherwise, I would just setup a new key-pair for the connection between "private" and "destination". — rich remer, Oct 04 '16 at 19:41

score 1 · Answer 1 · answered Oct 04 '16 at 20:10

ssh-add -l will list the keys added in your ssh-agent.

What makes you think you are connecting using your ssh-agent, when you write the key name on the command line? Before connecting to the first host, run

ssh-add my.key

Then you should be able to connect from the first hop to the target.

Zoredache · Answer 2 · 2016-10-04T19:22:56.697

0

What you should be looking for is a properly set SSH_AUTH_SOCK= environment variable on the intermediate systems after you have connected.

After conecting you should be able to run this command and get similar results. If you do not see an variable set, then forwarding didn't happen.

# export | grep SSH_AUTH
declare -x SSH_AUTH_SOCK="/tmp/ssh-oDOTx6u2AR/agent.47026"

Your server must permit forwarding, and your client must be configured to use forwarding. The server option AllowAgentForwarding defaults to yes, but could be disabled on your servers, if they have been hardened. On your client you can use the -A during connecting, or set the ForwardAgent option to yes. You should not do forwarding to untrusted hosts. So you might do something like this in your config.

# where *.example.org is pattern that matches all the systems you trust
# for forwarding purposes.
Host *.example.org
    ForwardAgent yes

Looking closer at your log, I can see that your client is offering the key, and the destination appears to not be accepting it.

debug2: key: me@me (0x5611e148efd0), ##  your key from the agent
debug2: key: /home/me/.ssh/id_rsa ((nil)),
debug2: key: /home/me/.ssh/id_dsa ((nil)),
debug2: key: /home/me/.ssh/id_ecdsa ((nil)),
...
debug1: Offering RSA public key: me@me
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug1: Authentications that can continue: publickey

edited Oct 04 '16 at 19:22

answered Oct 04 '16 at 19:12

Zoredache

130,897
41
276
420

Everything seems to be in order. I see SSH_AUTH_SOCK set on both "gateway" and "private". – rich remer Oct 04 '16 at 19:14
Looking closer at the logs, your agent appears to be offering the key. Are you sure your key is setup properly on the destination? – Zoredache Oct 04 '16 at 19:23
Yes, because I can connect directly to the destination using the key. Just not when forwarding through gateway and private hosts. – rich remer Oct 04 '16 at 19:42
Hrm, something weird is going on then. Your logs seem to clearly show the intermediate box offering a key, and it getting rejected. Your updated question seems to clearly show that there is a valid AGENT socket on the intermediate system. If you are sure that specific key works with the destination, I am not sure what the problem is. – Zoredache Oct 04 '16 at 19:45
1

The next step would probably be to increase the verbosity of logging on the destination SSH server, and look at the logs there. It seems likely that the logs on the destination server should tell you why it isn't accepting the key being offered by the intermediate system. – Zoredache Oct 04 '16 at 19:47

SSH agent forwarding fails depending on intermediaries

2 Answers2