Ldap ssh authentication is super slow... any way to speed it up?

Question

I am running OpenSUSE.

Here is the output of ssh -vvv:

OpenSSH_5.8p1, OpenSSL 1.0.0c 2 Dec 2010
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Applying options for *
debug2: ssh_connect: needpriv 0
debug1: Connecting to <ipaddress> [ipaddress] port 22.
debug1: Connection established.
debug1: permanently_set_uid: 0/0
debug3: Incorrect RSA1 identifier
debug3: Could not load "/root/.ssh/id_rsa" as a RSA1 public key
debug2: key_type_from_name: unknown key type '-----BEGIN'
debug3: key_read: missing keytype
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug3: key_read: missing whitespace
debug2: key_type_from_name: unknown key type '-----END'
debug3: key_read: missing keytype
debug1: identity file /root/.ssh/id_rsa type 1
debug1: identity file /root/.ssh/id_rsa-cert type -1
debug1: identity file /root/.ssh/id_dsa type -1
debug1: identity file /root/.ssh/id_dsa-cert type -1
debug1: identity file /root/.ssh/id_ecdsa type -1
debug1: identity file /root/.ssh/id_ecdsa-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.1
debug1: match: OpenSSH_5.1 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.8
debug2: fd 3 setting O_NONBLOCK
debug3: load_hostkeys: loading entries for host "ipaddress" from file "/root/.ssh/known_hosts"
debug3: load_hostkeys: found key type RSA in file /root/.ssh/known_hosts:4
debug3: load_hostkeys: loaded 1 keys
debug3: order_hostkeyalgs: prefer hostkeyalgs: ssh-rsa-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-rsa
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: kex_parse_kexinit: ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa-cert-v01@openssh.com,ssh-rsa-cert-v00@openssh.com,ssh-rsa,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-dss-cert-v01@openssh.com,ssh-dss-cert-v00@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-dss
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: none,zlib@openssh.com,zlib
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: first_kex_follows 0 
debug2: kex_parse_kexinit: reserved 0 
debug2: kex_parse_kexinit: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: kex_parse_kexinit: ssh-rsa,ssh-dss
debug2: kex_parse_kexinit: aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour128,arcfour256,arcfour,aes192-cbc,aes256-cbc,rijndael-cbc@lysator.liu.se,aes128-ctr,aes192-ctr,aes256-ctr
debug2: kex_parse_kexinit: aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,arcfour128,arcfour256,arcfour,aes192-cbc,aes256-cbc,rijndael-cbc@lysator.liu.se,aes128-ctr,aes192-ctr,aes256-ctr
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit: none,zlib@openssh.com
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: 
debug2: kex_parse_kexinit: first_kex_follows 0 
debug2: kex_parse_kexinit: reserved 0 
debug2: mac_setup: found hmac-md5
debug1: kex: server->client aes128-ctr hmac-md5 none
debug2: mac_setup: found hmac-md5
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug2: dh_gen_key: priv key bits set: 138/256
debug2: bits set: 529/1024
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Server host key: RSA cb:7f:ff:2e:65:28:f0:95:e6:8a:71:24:2a:67:02:2b
debug3: load_hostkeys: loading entries for host "<ipaddress>" from file "/root/.ssh/known_hosts"
debug3: load_hostkeys: found key type RSA in file /root/.ssh/known_hosts:4
debug3: load_hostkeys: loaded 1 keys
debug1: Host '<ipaddress>' is known and matches the RSA host key.
debug1: Found key in /root/.ssh/known_hosts:4
debug2: bits set: 504/1024
debug1: ssh_rsa_verify: signature correct
debug2: kex_derive_keys
debug2: set_newkeys: mode 1
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug2: set_newkeys: mode 0
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug2: key: /root/.ssh/id_rsa (0xb789d5c8)
debug2: key: /root/.ssh/id_dsa ((nil))
debug2: key: /root/.ssh/id_ecdsa ((nil))
debug1: Authentications that can continue: publickey,keyboard-interactive
debug3: start over, passed a different list publickey,keyboard-interactive
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /root/.ssh/id_rsa
debug3: send_pubkey_test
debug2: we sent a publickey packet, wait for reply

It hangs here for a good 30 seconds to a minute then

debug1: Authentications that can continue: publickey,keyboard-interactive
debug1: Trying private key: /root/.ssh/id_dsa
debug3: no such identity: /root/.ssh/id_dsa
debug1: Trying private key: /root/.ssh/id_ecdsa
debug3: no such identity: /root/.ssh/id_ecdsa
debug2: we did not send a packet, disable method
debug3: authmethod_lookup keyboard-interactive
debug3: remaining preferred: password
debug3: authmethod_is_enabled keyboard-interactive
debug1: Next authentication method: keyboard-interactive
debug2: userauth_kbdint
debug2: we sent a keyboard-interactive packet, wait for reply
debug2: input_userauth_info_req
debug2: input_userauth_info_req: num_prompts 1

I added PubkeyAuthentication no to the /etc/ssh/ssh_config and the /etc/ssh/sshd_config which makes it faster getting to the password prompt, but the password prompt still takes some time. Any way to fix that?

Here is where the password hangs

debug3: packet_send2: adding 32 (len 25 padlen 7 extra_pad 64)
debug2: input_userauth_info_req
debug2: input_userauth_info_req: num_prompts 0
debug3: packet_send2: adding 48 (len 10 padlen 6 extra_pad 64)
debug1: Authentication succeeded (keyboard-interactive).
Authenticated to ipaddress ([ipaddress]:22).
debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug1: Requesting no-more-sessions@openssh.com
debug1: Entering interactive session.

FIXED!!!!!!!!!!!!!! What is did... In the nsswitch_conf I had ldap included in the group and passwd which slows it down a lot. Thank you everybody for your input

passwd: compat
group:  files 

hosts:  files dns
networks:       files dns

tried turning off reverse lookups in the sshd config ? that'd be my first thing to try. — Sirex, Aug 15 '11 at 12:43
Anything in the logs? Any way to get a network traffic dump of what's going on in the pause? SSH debug mode output? Anything in the LDAP server logs, or sniffing traffic on the LDAP port of what's happening from your client's IP? — Bart Silverstrim, Aug 15 '11 at 13:01
I added useDNS no in the /etc/sh/ssh_config and that didn't seem to help either — Johnathon, Aug 15 '11 at 13:04
there is nothing in the logs that I can see. The /var/log/messages is clean — Johnathon, Aug 15 '11 at 13:06
Did you try doing the useDNS no bit in the sshd_config file on the server side? We had a DNS server on the server side go down, which caused the exact same symptoms and pretty much the exact same verbose output. Getting the DNS server back up (or turn useDNS off) was the solution. — Safado, Aug 15 '11 at 14:19
@johnathon Please add your answer to this question and accept it so it doesn't appear in the unanswered lists. — Paul, Aug 16 '11 at 12:48

cweiske · Answer 1 · 2011-08-15T13:06:45.483

1

Your ssh client might be trying GSSAPIAuthentication, which takes 30 seconds to timeout. You can disable that in your .ssh/config file with

Host example.org
 GSSAPIAuthentication no

If that does not help, use -v or -vvv to get debug information.

When it hangs at public key authentication - and you don't use public keys - also disable that:

PubkeyAuthentication no

edited Aug 15 '11 at 13:06

answered Aug 15 '11 at 12:47

cweiske

791
1
13
36

That is already commented out but I uncommented and it still is slow... – Johnathon Aug 15 '11 at 12:54
I just added that, restarted ssh and it still hangs at the "we sent a public key packet, wait for reply" – Johnathon Aug 15 '11 at 13:11
I mean ~/.ssh/config - not the sshD config – cweiske Aug 15 '11 at 13:28
all that is under ~/.ssh is known hosts this is the file that I was editing vi /etc/ssh/ssh_config – Johnathon Aug 15 '11 at 13:32
Then create the file. – cweiske Aug 15 '11 at 13:54
I had to add it to the sshd_config file as well, It asks for the password prompt right away now but after you enter the password it still hangs for a bit. – Johnathon Aug 15 '11 at 14:13

score 0 · Answer 2 · answered Oct 04 '22 at 15:49

There are a handful of issues which might cause slowness in ssh / sudo authentications.

Most recently, we were experiencing really slow SSH authentications for Active Directory logins on a Realm domain joined Linux host, where the domain controller was on a separate VLAN.

Almost anything which required domain authentication was really slow and caused a lot of frustration.

To troubleshoot we ran port scan, helping us determine that the LDAP query across VLAN was passing through our firewall but a firewall flag to allow LDAP was not properly configured. Additionally this was attempting to run over port 3268 for Global Catalogue which would always timeout (hence the really long login times).

We got to this solution based on a helpful reddit thread with various suggestions here

score 0 · Answer 3 · answered Jun 01 '12 at 01:50

I have previously experienced something similar that was a result of setting the search scope to "sub" rather than scope of "base", because sub will do a full tree search if you set the base to your root like "dc=example,dc=com", and the scope to -s sub

If you try the test from the command line with -s set to base rather than sub you get a much faster query...

 time ldapsearch -D "cn=directory manager" -w secret -p 389 \
 -h server.example.com -b "dc=example,dc=com" -s sub "(objectclass=*)"

and then try again with the scope as base

time ldapsearch -D "cn=directory manager" -w secret -p 389 \
-h server.example.com -b "" -s base "objectclass=*"

Ldap ssh authentication is super slow... any way to speed it up?

3 Answers3