1. Problem Summary:
Hi, recently, in our infrastructure production environment, new users are unable to ssh into a debian jump server. The jumpserver' SSH daemon retrieves the SSH public keys from the LDAP Server. We verified sshd local configuration, sshd logs, LDAP public keys presence and we are unable to fully understand what is preventing the users to effectively login.
2. Details (Configuration Files):
# Package generated configuration file
# See the sshd_config(5) manpage for details
# What ports, IPs and protocols we listen for
Port 22
# Use these options to restrict which interfaces/protocols sshd will bind to
#ListenAddress ::
#ListenAddress 0.0.0.0
Protocol 2
# HostKeys for protocol version 2
HostKey /etc/ssh/ssh_host_rsa_key
HostKey /etc/ssh/ssh_host_dsa_key
HostKey /etc/ssh/ssh_host_ecdsa_key
HostKey /etc/ssh/ssh_host_ed25519_key
#Privilege Separation is turned on for security
UsePrivilegeSeparation yes
# Logging
SyslogFacility AUTH
LogLevel VERBOSE
# Authentication:
LoginGraceTime 120
PermitRootLogin no
StrictModes yes
PubkeyAuthentication yes
# Don't read the user's ~/.rhosts and ~/.shosts files
IgnoreRhosts yes
# similar for protocol version 2
HostbasedAuthentication no
# Uncomment if you don't trust ~/.ssh/known_hosts for RhostsRSAAuthentication
#IgnoreUserKnownHosts yes
# To enable empty passwords, change to yes (NOT RECOMMENDED)
PermitEmptyPasswords no
# Change to yes to enable challenge-response passwords (beware issues with
# some PAM modules and threads)
ChallengeResponseAuthentication no
# Change to no to disable tunnelled clear text passwords
PasswordAuthentication no
# Kerberos options
#KerberosAuthentication no
#KerberosGetAFSToken no
#KerberosOrLocalPasswd yes
#KerberosTicketCleanup yes
# GSSAPI options
#GSSAPIAuthentication no
#GSSAPICleanupCredentials yes
X11Forwarding no
PrintMotd no
PrintLastLog yes
TCPKeepAlive yes
#UseLogin no
#MaxStartups 10:30:60
#Banner /etc/issue.net
# Allow client to pass locale environment variables
AcceptEnv LANG LC_*
Subsystem sftp /usr/lib/openssh/sftp-server
# Set this to 'yes' to enable PAM authentication, account processing,
# and session processing. If this is enabled, PAM authentication will
# be allowed through the ChallengeResponseAuthentication and
# PasswordAuthentication. Depending on your PAM configuration,
# PAM authentication via ChallengeResponseAuthentication may bypass
# the setting of "PermitRootLogin without-password".
# If you just want the PAM account and session checks to run without
# PAM authentication, then enable this but set PasswordAuthentication
# and ChallengeResponseAuthentication to 'no'.
UsePAM yes
UseDNS no
Match Address 172.20.1.*,10.*
PasswordAuthentication yes
Match Group ldap-user
AuthorizedKeysFile /dev/null
AuthorizedKeysCommand /opt/ourcompanyname/scripts/ldap_keys.sh
AuthorizedKeysCommandUser nobody
Script which queries the LDAP:
#!/bin/bash
if [ $(whoami) = nobody ]; then
/usr/local/bin/ldap_ssh_keys -f /tmp/ssh_keys.log -b dc=prod,dc=ourcompany -u ldaps://ldap.prod.ourcompanydomain -l DEBUG $1
else
echo "should be run as user nobody"
fi
3. What we tried (logs):
User tries to connect to the server (client side)
myusername@laptophostname currentfoldr % ssh -vvv myusername@jbox01.prod.ourcompanydomain
OpenSSH_7.9p1, LibreSSL 2.7.3
debug1: Reading configuration data /Users/myusername/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 48: Applying options for *
debug2: resolving "jbox01.prod.ourcompanydomain" port 22
debug2: ssh_connect_direct
debug1: Connecting to jbox01.prod.ourcompanydomain [172.20.1.66] port 22.
debug1: Connection established.
debug1: identity file /Users/myusername/.ssh/id_rsa type 0
debug1: identity file /Users/myusername/.ssh/id_rsa-cert type -1
debug1: identity file /Users/myusername/.ssh/id_dsa type -1
debug1: identity file /Users/myusername/.ssh/id_dsa-cert type -1
debug1: identity file /Users/myusername/.ssh/id_ecdsa type -1
debug1: identity file /Users/myusername/.ssh/id_ecdsa-cert type -1
debug1: identity file /Users/myusername/.ssh/id_ed25519 type -1
debug1: identity file /Users/myusername/.ssh/id_ed25519-cert type -1
debug1: identity file /Users/myusername/.ssh/id_xmss type -1
debug1: identity file /Users/myusername/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.9
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4p1 Debian-10+deb9u6
debug1: match: OpenSSH_7.4p1 Debian-10+deb9u6 pat OpenSSH_7.0*,OpenSSH_7.1*,OpenSSH_7.2*,OpenSSH_7.3*,OpenSSH_7.4*,OpenSSH_7.5*,OpenSSH_7.6*,OpenSSH_7.7* compat 0x04000002
debug2: fd 5 setting O_NONBLOCK
debug1: Authenticating to jbox01.prod.ourcompanydomain:22 as 'myusername'
debug3: hostkeys_foreach: reading file "/Users/myusername/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /Users/myusername/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from jbox01.prod.ourcompanydomain
debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1
debug2: host key algorithms: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:VKd849z2c6XlUO3p7GnZpwVIwtdhI6Gl+6EsTImYwLI
debug3: hostkeys_foreach: reading file "/Users/myusername/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /Users/myusername/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from jbox01.prod.ourcompanydomain
debug3: hostkeys_foreach: reading file "/Users/myusername/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /Users/myusername/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from 172.20.1.66
debug1: Host 'jbox01.prod.ourcompanydomain' is known and matches the ECDSA host key.
debug1: Found key in /Users/myusername/.ssh/known_hosts:1
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey after 134217728 blocks
debug1: Will attempt key: /Users/myusername/.ssh/id_rsa RSA SHA256:6pLYo9k0m0LK2E4Og69uAmu24SyTOX8FUX91/HDx1gk
debug1: Will attempt key: /Users/myusername/.ssh/id_dsa
debug1: Will attempt key: /Users/myusername/.ssh/id_ecdsa
debug1: Will attempt key: /Users/myusername/.ssh/id_ed25519
debug1: Will attempt key: /Users/myusername/.ssh/id_xmss
debug2: pubkey_prepare: done
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /Users/myusername/.ssh/id_rsa RSA SHA256:6pLYo9k0m0LK2E4Og69uAmu24SyTOX8FUX91/HDx1gk
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug1: Trying private key: /Users/myusername/.ssh/id_dsa
debug3: no such identity: /Users/myusername/.ssh/id_dsa: No such file or directory
debug1: Trying private key: /Users/myusername/.ssh/id_ecdsa
debug3: no such identity: /Users/myusername/.ssh/id_ecdsa: No such file or directory
debug1: Trying private key: /Users/myusername/.ssh/id_ed25519
debug3: no such identity: /Users/myusername/.ssh/id_ed25519: No such file or directory
debug1: Trying private key: /Users/myusername/.ssh/id_xmss
debug3: no such identity: /Users/myusername/.ssh/id_xmss: No such file or directory
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
myusername@jbox01.prod.ourcompanydomain: Permission denied (publickey).
myusername@mylaptopname:currentfoldername % ssh -vv myusername@jbox01.prod.ourcompanydomain
OpenSSH_7.9p1, LibreSSL 2.7.3
debug1: Reading configuration data /Users/myusername/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 48: Applying options for *
debug2: resolving "jbox01.prod.ourcompanydomain" port 22
debug2: ssh_connect_direct
debug1: Connecting to jbox01.prod.ourcompanydomain [172.20.1.66] port 22.
debug1: Connection established.
debug1: identity file /Users/myusername/.ssh/id_rsa type 0
debug1: identity file /Users/myusername/.ssh/id_rsa-cert type -1
debug1: identity file /Users/myusername/.ssh/id_dsa type -1
debug1: identity file /Users/myusername/.ssh/id_dsa-cert type -1
debug1: identity file /Users/myusername/.ssh/id_ecdsa type -1
debug1: identity file /Users/myusername/.ssh/id_ecdsa-cert type -1
debug1: identity file /Users/myusername/.ssh/id_ed25519 type -1
debug1: identity file /Users/myusername/.ssh/id_ed25519-cert type -1
debug1: identity file /Users/myusername/.ssh/id_xmss type -1
debug1: identity file /Users/myusername/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.9
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4p1 Debian-10+deb9u6
debug1: match: OpenSSH_7.4p1 Debian-10+deb9u6 pat OpenSSH_7.0*,OpenSSH_7.1*,OpenSSH_7.2*,OpenSSH_7.3*,OpenSSH_7.4*,OpenSSH_7.5*,OpenSSH_7.6*,OpenSSH_7.7* compat 0x04000002
debug2: fd 5 setting O_NONBLOCK
debug1: Authenticating to jbox01.prod.ourcompanydomain:22 as 'myusername'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1
debug2: host key algorithms: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:VKd849z2c6XlUO3p7GnZpwVIwtdhI6Gl+6EsTImYwLI
debug1: Host 'jbox01.prod.ourcompanydomain' is known and matches the ECDSA host key.
debug1: Found key in /Users/myusername/.ssh/known_hosts:1
debug2: set_newkeys: mode 1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey after 134217728 blocks
debug1: Will attempt key: /Users/myusername/.ssh/id_rsa RSA SHA256:6pLYo9k0m0LK2E4Og69uAmu24SyTOX8FUX91/HDx1gk
debug1: Will attempt key: /Users/myusername/.ssh/id_dsa
debug1: Will attempt key: /Users/myusername/.ssh/id_ecdsa
debug1: Will attempt key: /Users/myusername/.ssh/id_ed25519
debug1: Will attempt key: /Users/myusername/.ssh/id_xmss
debug2: pubkey_prepare: done
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
**debug1: Offering public key: /Users/myusername/.ssh/id_rsa RSA SHA256:6pLYo9k0m0LK2E4Og69uAmu24SyTOX8FUX91/HDx1gk
debug2: we sent a publickey packet, wait for reply**
debug1: Authentications that can continue: publickey
debug1: Trying private key: /Users/myusername/.ssh/id_dsa
debug1: Trying private key: /Users/myusername/.ssh/id_ecdsa
debug1: Trying private key: /Users/myusername/.ssh/id_ed25519
debug1: Trying private key: /Users/myusername/.ssh/id_xmss
**debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
myusername@jbox01.prod.ourcompanydomain: Permission denied (publickey).**
User tries to connect to the server (server side):
Mar 16 12:49:55 jbox01 sshd[26063]: Connection from 172.20.101.6 port 63442 on 172.20.1.66 port 22
Mar 16 12:49:56 jbox01 sshd[26063]: Failed publickey for myusername from 172.20.101.6 port 63442 ssh2: RSA SHA256:6pLYo9k0m0LK2E4Og69uAmu24SyTOX8FUX91/HDx1gk
We have also compared the SHA256 of the key contained in LDAP with the key the user is providing to the server and they match. Any suggestion on this ?
