4

I am trying to connect to an SSH server I was able to connect to in the past but using different OpenSSH client versions.

Starting from a Cygwin OpenSSH_6.6.1, OpenSSL 1.0.1g 7 Apr 2014 client, going through a Squid Proxy using corkscrew, I am connecting fine to an intermediate Ubuntu 13.10 OpenSSH_6.2p2 Ubuntu-6ubuntu0.3 server using a forwarded agent to bypass a firewall that prevents direct access to the target server.

I am able to connect fine to other SSH servers through this intermediate server which rules out forwarded agent configuration or permissions issues.

The only difference I can tell with the target server that fails is that it runs an old Debian version using OpenSSH_4.3 that has not been updated for several years although I was able to connect to it a few months ago (before upgrading the intermediate server following the heartbleed bug) and before upgrading my Cygwin client.

Are there any known protocol incompatibility issues between recent and some older OpenSSH user-agents?

Connecting from the intermediate server to the target server (s1.hidden.com @63.82.7.10 in the traces bellow) fails without any error message and after the target server has reported to accept my forwarded public key and the intermediate server has sent that public key.

In the trace bellow both the connection to the intermediate server and the attempted connection to the target server have options -vvv which explains channel 1 debug messages involving the forwarded agent (addresses are edited):

[...]
debug2: we sent a publickey packet, wait for reply
debug1: Server accepts key: pkalg ssh-rsa blen 279
debug2: input_userauth_pk_ok: fp 46:83:33:00:28:48:b2:3c:b1:3f:60:bb:62:80:cc:d2
debug3: sign_and_send_pubkey: RSA 46:83:33:00:28:48:b2:3c:b1:3f:60:bb:62:80:cc:d2
debug2: channel 1: rcvd eof
debug2: channel 1: output open -> drain
debug2: channel 1: obuf empty
debug2: channel 1: close_write
debug2: channel 1: output drain -> closed
debug1: channel 1: FORCE input drain
debug2: channel 1: ibuf empty
debug2: channel 1: send eof
debug2: channel 1: input drain -> closed
debug2: channel 1: send close
Connection closed by 63.82.7.10
root@intermediate-server: ~
$ debug3: channel 1: will not send data after close
debug2: channel 1: rcvd close
debug3: channel 1: will not send data after close
debug2: channel 1: is dead
debug2: channel 1: garbage collecting
debug1: channel 1: free: authentication agent connection, nchannels 2
debug3: channel 1: status: The following connections are open:
  #0 client-session (t4 r0 i0/0 o0/0 fd 3/6 cc -1)
  #1 authentication agent connection (t4 r2 i3/0 o3/0 fd 8/8 cc -1)

On a target server that succeeds I can see:

debug2: channel 1: input drain -> closed
debug2: channel 1: send close
debug1: Authentication succeeded (publickey).

Instead of this on the failed server:

debug2: channel 1: input drain -> closed
debug2: channel 1: send close
Connection closed by 63.82.7.10

Everything else appears to be identical other than OpenSSH versions and addresses.

Here is the full -vvv trace from the intermediate server to the target server:

OpenSSH_6.2p2 Ubuntu-6ubuntu0.3, OpenSSL 1.0.1e 11 Feb 2013
debug1: Reading configuration data /root/.ssh/config
debug1: /root/.ssh/config line 1: Applying options for s1
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to s1.hidden.com [63.82.7.10] port 2222.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug1: identity file /root/.ssh/id_rsa type -1
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: identity file /root/.ssh/id_dsa type -1
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.2p2 Ubuntu-6ubuntu0.3
debug1: Remote protocol version 2.0, remote software version OpenSSH_4.3
debug1: match: OpenSSH_4.3 pat OpenSSH_4*
debug2: fd 3 setting O_NONBLOCK
debug3: put_host_port: [s1.hidden.com]:2222
debug3: load_hostkeys: loading entries for host "[s1.hidden.com]:2222" from file "/root/.ssh/known_hosts"
debug3: load_hostkeys: found key type RSA in file /root/.ssh/known_hosts:4
debug3: load_hostkeys: loaded 1 keys
debug3: order_hostkeyalgs: prefer hostkeyalgs: ssh-rsa-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-rsa
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-rsa,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-dss-cert-v00@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-ripemd160-etm@openssh.com,hmac-sha1-96-etm@openssh.com,hmac-md5-96-etm@openssh.com,hmac-md5,hmac-sha1,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour128,arcfour256,arcfour,aes192-cbc,aes256-cbc,rijndael-cbc@lysator.liu.se,aes128-ctr,aes192-ctr,aes256-ctr
debug2: kex_parse_kexinit: aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour128,arcfour256,arcfour,aes192-cbc,aes256-cbc,rijndael-cbc@lysator.liu.se,aes128-ctr,aes192-ctr,aes256-ctr
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit:
debug2: kex_parse_kexinit: first_kex_follows 0
debug2: kex_parse_kexinit: reserved 0
debug2: mac_setup: found hmac-md5
debug1: kex: server->client aes128-ctr hmac-md5 none
debug2: mac_setup: found hmac-md5
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug2: dh_gen_key: priv key bits set: 135/256
debug2: bits set: 507/1024
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Server host key: RSA b0:07:f4:ce:c7:00:e2:54:23:1c:45:7f:d9:61:41:8a
debug3: put_host_port: [63.82.7.10]:2222
debug3: put_host_port: [s1.hidden.com]:2222
debug3: load_hostkeys: loading entries for host "[s1.hidden.com]:2222" from file "/root/.ssh/known_hosts"
debug3: load_hostkeys: found key type RSA in file /root/.ssh/known_hosts:4
debug3: load_hostkeys: loaded 1 keys
debug3: load_hostkeys: loading entries for host "[63.82.7.10]:2222" from file "/root/.ssh/known_hosts"
debug3: load_hostkeys: found key type RSA in file /root/.ssh/known_hosts:5
debug3: load_hostkeys: loaded 1 keys
debug1: Host '[s1.hidden.com]:2222' is known and matches the RSA host key.
debug1: Found key in /root/.ssh/known_hosts:4
debug2: bits set: 535/1024
debug1: ssh_rsa_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: client_input_channel_open: ctype auth-agent@openssh.com rchan 2 win 65536 max 16384
debug2: fd 8 setting O_NONBLOCK
debug3: fd 8 is O_NONBLOCK
debug1: channel 1: new [authentication agent connection]
debug1: confirm auth-agent@openssh.com
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /home/jvincent/.ssh/id_rsa (0xb8485070),
debug2: key: /root/.ssh/id_rsa ((nil)),
debug2: key: /root/.ssh/id_dsa ((nil)),
debug2: key: /root/.ssh/id_ecdsa ((nil)),
debug1: Authentications that can continue: publickey,password
debug3: start over, passed a different list publickey,password
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/jvincent/.ssh/id_rsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply
debug1: Server accepts key: pkalg ssh-rsa blen 279
debug2: input_userauth_pk_ok: fp 46:83:33:00:28:48:b2:3c:b1:3f:60:bb:62:80:cc:d2
debug3: sign_and_send_pubkey: RSA 46:83:33:00:28:48:b2:3c:b1:3f:60:bb:62:80:cc:d2
debug2: channel 1: rcvd eof
debug2: channel 1: output open -> drain
debug2: channel 1: obuf empty
debug2: channel 1: close_write
debug2: channel 1: output drain -> closed
debug1: channel 1: FORCE input drain
debug2: channel 1: ibuf empty
debug2: channel 1: send eof
debug2: channel 1: input drain -> closed
debug2: channel 1: send close
Connection closed by 63.82.7.10
root@intermediate-server: ~
$ debug3: channel 1: will not send data after close
debug2: channel 1: rcvd close
debug3: channel 1: will not send data after close
debug2: channel 1: is dead
debug2: channel 1: garbage collecting
debug1: channel 1: free: authentication agent connection, nchannels 2
debug3: channel 1: status: The following connections are open:
  #0 client-session (t4 r0 i0/0 o0/0 fd 3/6 cc -1)
  #1 authentication agent connection (t4 r2 i3/0 o3/0 fd 8/8 cc -1)
Axel Beckert
  • 398
  • 2
  • 17
Jean Vincent
  • 271
  • 3
  • 6
  • 2
    Are there any intersting/relevant messages in the server's logs ? – user9517 May 24 '14 at 09:06
  • Do you have a exit in your shell ? Or do you have a shell push to /bin/false in /etc/passwd. But check the logs ! – Dom May 24 '14 at 09:54
  • I do not have access to that server other than through ssh at this time, so I cannot see the logs. I don't believe anyone would have modified the shell since I last logged-in. The only difference seems to be the upgrade of both my client and the intermediate server. – Jean Vincent May 24 '14 at 11:04
  • Put a KVM tail on the box. I bet there are interesting things in the logs. Otherwise you can start sshd on the server with -d and it won't daemonize so you can see what's going on. – Andrew Domaszek May 24 '14 at 12:17
  • At do not have a physical access to this server. The administrator of the server may not be available until Monday. – Jean Vincent May 24 '14 at 12:29
  • I now have access to the server, using a password, the logs show nothing unusual, no trace of failed connections attempts. No partition is full. I have also tried to connect from the intermediate server using a local identity, it also fails. – Jean Vincent May 25 '14 at 10:49

1 Answers1

1

The way to debug these issues is to get a secondary channel/session to the server, and then start `which sshd` -d -p 2222 and check the output of the sshd to get better information. The guilty in my experiences:

  • bad shell, check /etc/passwd for the user, and check the permissions and existence of the shell for the user listed
  • check the authorized_keys for forced commands on the public key
  • errors in .profile ._rc .*env etc.
  • bad permissions on $HOME $HOME/.ssh and $HOME/.ssh/authorized_keys, sshd can be pedantic with that. (However, I don't suspect this one, as it apepared the servers accepted the keys)

NOTE: you appears to have "ssh -vv" running to the intermediary server, as the debug*: messages after the $ prompt indicates the intermediary -> target exited, but now you have some source->intermediary verbose/debugging also interleaved with the debug information you are looking for/at.

Hvisage
  • 386
  • 2
  • 7