3

I've been ssh'ing to this box without problem for a year using public-key authentication.

No changes have been made to the client or server's ssh config. Now, ssh connections never complete or fail, they just hang. The following is the debug output when trying to connect:

OpenSSH_6.0p1 Debian-3ubuntu1, OpenSSL 1.0.1c 10 May 2012
debug1: Reading configuration data /home/jivan/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to <host> [<ip>] port <port>.
debug1: Connection established.
debug1: identity file .ssh/id_rsa.pub type 1
debug1: Checking blacklist file /usr/share/ssh/blacklist.RSA-2048
debug1: Checking blacklist file /etc/ssh/blacklist.RSA-2048
debug1: identity file .ssh/id_rsa.pub-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.8p1 Debian-7ubuntu1
debug1: match: OpenSSH_5.8p1 Debian-7ubuntu1 pat OpenSSH_5*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.0p1 Debian-3ubuntu1
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ECDSA 81:63:05:0c:39:6b:77:d7:a2:25:93:30:80:f1:ba:0e
debug1: Host '[<host>]:<port>' is known and matches the ECDSA host key.
debug1: Found key in /home/jivan/.ssh/known_hosts:39
debug1: ssh_ecdsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Offering RSA public key: .ssh/id_rsa.pub
debug1: Server accepts key: pkalg ssh-rsa blen 279

Server log:

Feb 24 12:02:38 balance sshd[5734]: debug1: Forked child 6745.
Feb 24 12:02:38 balance sshd[6745]: Set /proc/self/oom_score_adj to 0
Feb 24 12:02:38 balance sshd[6745]: debug1: rexec start in 5 out 5 newsock 5 pip
e 7 sock 8
Feb 24 12:02:38 balance sshd[6745]: debug1: inetd sockets after dupping: 3, 3
Feb 24 12:02:38 balance sshd[6745]: Connection from 207.204.228.55 port 57967
Feb 24 12:02:38 balance sshd[6745]: debug1: Client protocol version 2.0; client 
software version OpenSSH_6.0p1 Debian-3ubuntu1
Feb 24 12:02:38 balance sshd[6745]: debug1: match: OpenSSH_6.0p1 Debian-3ubuntu1
 pat OpenSSH*
Feb 24 12:02:38 balance sshd[6745]: debug1: Enabling compatibility mode for prot
ocol 2.0
Feb 24 12:02:38 balance sshd[6745]: debug1: Local version string SSH-2.0-OpenSSH_5.8p1 Debian-7ubuntu1
Feb 24 12:02:39 balance sshd[6745]: debug1: PAM: initializing for "jivan"
Feb 24 12:02:39 balance sshd[6745]: debug1: PAM: setting PAM_RHOST to "207.204.228.55"
Feb 24 12:02:39 balance sshd[6745]: debug1: PAM: setting PAM_TTY to "ssh"
Feb 24 12:02:40 balance sshd[6745]: debug1: Checking blacklist file /usr/share/ssh/blacklist.RSA-2048
Feb 24 12:02:40 balance sshd[6745]: debug1: Checking blacklist file /etc/ssh/blacklist.RSA-2048
Feb 24 12:02:40 balance sshd[6745]: debug1: temporarily_use_uid: 1000/1000 (e=0/0)
Feb 24 12:02:40 balance sshd[6745]: debug1: trying public key file /home/jivan/.ssh/authorized_keys
Feb 24 12:02:40 balance sshd[6745]: debug1: fd 4 clearing O_NONBLOCK
Feb 24 12:02:40 balance sshd[6745]: debug1: matching key found: file /home/jivan/.ssh/authorized_keys, line 1
Feb 24 12:02:40 balance sshd[6745]: Found matching RSA key: 5f:8c:76:ad:48:6a:f5:73:03:6a:36:13:c0:ea:3b:45
Feb 24 12:02:40 balance sshd[6745]: debug1: restore_uid: 0/0

Any ideas what the problem could be?

Update: The problem stopped as mysteriously as it started. Still interested in any insight as to what might have caused this.

JivanAmara
  • 171
  • 1
  • 6
  • It would be more helpful to see the log of the server. To get it you should change the `LogLevel` directive in `/etc/ssh/sshd_config` on the SSH server, then restart it. Then try reconnecting again and check the server logs. – replay Feb 24 '13 at 11:59
  • Did you give it a few minutes to finish? It could have been your DNS server was down and it was waiting for the reverse DNS to time out – R. S. Feb 25 '13 at 01:19
  • I think I saw a similar problem caused by a broken *openssl.cnf* on the server-side but this is just a wild guess in your case. – scai Feb 25 '13 at 07:26

4 Answers4

3

This is usually caused by an issue with reverse DNS; just had this happen to one of my legacy servers the other week. You were right to invoke sshd -d, but the reverse DNS problems won't really become apparent unless you run it with sshd -ddd instead so that you're seeing the output of debug3.

While reverse DNS issues don't always cause things to back up this badly, if enough DNS lookups related to the connection attempt are failing (i.e. all DNS servers are completely unavailable, or enough of the related zones/reverse zones are timing out), under some configurations it will be enough to completely lock out ssh access to the system. It doesn't matter how high you set your client timeout if the server gets impatient and drops the connection.

Andrew B
  • 32,588
  • 12
  • 93
  • 131
  • I think that `-d` isn't valid flag on Debian. Did you mean `-v`? Running `ssh -vv hostname` solves the issue for me. – Tombart Sep 23 '16 at 11:24
  • @Tombart Good catch, I was referring to `sshd`. (which uses `-d` instead of `-v`) – Andrew B Sep 23 '16 at 16:49
1

I've had this problem too, but here it was the client's turn to talk. In this case, it appeared to be due to the presence of a gpg-agent (running in ssh-agent-support mode); killing it or ssh-add -D'ing it allowed resumption of ssh connections.

fche
  • 301
  • 2
  • 6
  • I just ran into this problem. Killing `ssh-agent`, restarting and re-doing `ssh-add` made it go away, making this a useful answer. It was not reverse DNS. – Kaz Jun 05 '17 at 22:07
1

In my case, it was caused by a corrupt keyring. If one runs with more debugging info (ssh -vvv), one will get:

debug1: Offering RSA public key: /home/user/.ssh/id_rsa
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 60
debug1: Server accepts key: pkalg rsa-sha2-512 blen 535
debug2: input_userauth_pk_ok: fp
SHA256:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
debug3: sign_and_send_pubkey: RSA
SHA256:XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX

Per this, I got the hint it was related to the SSH_AUTH_SOCK variable. Then I located it:

user@user-office / $ $SSH_AUTH_SOCK
bash: /run/user/1000/keyring/ssh: No such device or address

I then deleted this file and reran the ssh command. This caused the pass prompt to appear and then it worked.

CoderGuy123
  • 321
  • 1
  • 3
  • 11
0

While the situation above is most likely DNS related, I kept experiencing stalls right after 

debug1: expecting SSH2_MSG_KEX_ECDH_REPLY

it was not consistent either. It went away by doing:

KexAlgorithms diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1

I suspect the real origin is limited MTU of IPv6 tunnel, but I have yet to verify this. Google finds this question.

mcr
  • 233
  • 1
  • 10