0

I've tried setting up passwordless sshing for a MPI cluster im working on. Im generating the RSA key on the master node and sending the public key to my client node. It sends but refuses to connect via the public/private keys and falls back to connecting via password when I try to connect via passwordless sshing. Here is output from debug and log files, any help would be appreciated!

client /var/log/auth.log

Oct 14 16:08:08 mpiuser1 sshd[7249]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=192.168.1.100  user=mpiuser

Master .ssh perms

drwx------  2 mpiuser mpiuser   4096 Oct 14 15:08 .ssh

inside ssh dir
drwx------  2 mpiuser mpiuser 4096 Oct 14 15:08 .
drwx------ 19 mpiuser mpiuser 4096 Oct 14 16:04 ..
-rw-------  1 mpiuser mpiuser 1679 Oct 14 15:07 id_rsa
-rw-r--r--  1 mpiuser mpiuser  399 Oct 14 15:07 id_rsa.pub
-rw-r--r--  1 mpiuser mpiuser  444 Oct 14 15:05 known_hosts

client ssh perms

drwx------ 2 mpiuser mpiuser 4096 Oct 14 15:09 .ssh

inside ssh dir
drwx------ 2 mpiuser mpiuser 4096 Oct 14 16:27 .
drwx------ 5 mpiuser mpiuser 4096 Oct  9 15:14 ..
-rw------- 1 mpiuser mpiuser  615 Oct 14 15:09 authorized_keys
-rw------- 1 mpiuser mpiuser 1675 Oct 14 16:27 id_rsa
-rw-r--r-- 1 mpiuser mpiuser  398 Oct 14 16:27 id_rsa.pub
-rw-r--r-- 1 mpiuser mpiuser  444 Oct 14 15:06 known_hosts

ssh debug read-out when connecting from master -> client

mpiuser@mpiMaster:~$ ssh -vvv mpiuser@mpiuser1
OpenSSH_7.2p2 Ubuntu-4ubuntu2.8, OpenSSL 1.0.2g  1 Mar 2016
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: resolving "mpiuser1" port 22
debug2: ssh_connect_direct: needpriv 0
debug1: Connecting to mpiuser1 [192.168.1.101] port 22.
debug1: Connection established.
debug1: identity file /home/mpiuser/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /home/mpiuser/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/mpiuser/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/mpiuser/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/mpiuser/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/mpiuser/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/mpiuser/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /home/mpiuser/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.2p2 Ubuntu-4ubuntu2.8
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.2p2 Ubuntu-4ubuntu2.8
debug1: match: OpenSSH_7.2p2 Ubuntu-4ubuntu2.8 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to mpiuser1:22 as 'mpiuser'
debug3: hostkeys_foreach: reading file "/home/mpiuser/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/mpiuser/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from mpiuser1
debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,3des-cbc
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com,aes128-cbc,aes192-cbc,aes256-cbc,3des-cbc
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group14-sha1
debug2: host key algorithms: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:iNrg1vpxjB7F4kJhdcqTwx983ivFKhXTDjqyQocnuKk
debug3: hostkeys_foreach: reading file "/home/mpiuser/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/mpiuser/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from mpiuser1
debug3: hostkeys_foreach: reading file "/home/mpiuser/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/mpiuser/.ssh/known_hosts:2
debug3: load_hostkeys: loaded 1 keys from 192.168.1.101
debug1: Host 'mpiuser1' is known and matches the ECDSA host key.
debug1: Found key in /home/mpiuser/.ssh/known_hosts:1
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey after 134217728 blocks
debug1: Skipping ssh-dss key /home/mpiuser/.ssh/id_dsa - not in PubkeyAcceptedKeyTypes
debug2: key: /home/mpiuser/.ssh/id_rsa (0x55f66d882150)
debug2: key: /home/mpiuser/.ssh/id_dsa ((nil))
debug2: key: /home/mpiuser/.ssh/id_ecdsa ((nil))
debug2: key: /home/mpiuser/.ssh/id_ed25519 ((nil))
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<rsa-sha2-256,rsa-sha2-512>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password
debug3: start over, passed a different list publickey,password
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /home/mpiuser/.ssh/id_rsa
debug3: send_pubkey_test
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,password
debug1: Trying private key: /home/mpiuser/.ssh/id_dsa
debug3: no such identity: /home/mpiuser/.ssh/id_dsa: No such file or directory
debug1: Trying private key: /home/mpiuser/.ssh/id_ecdsa
debug3: no such identity: /home/mpiuser/.ssh/id_ecdsa: No such file or directory
debug1: Trying private key: /home/mpiuser/.ssh/id_ed25519
debug3: no such identity: /home/mpiuser/.ssh/id_ed25519: No such file or directory
debug2: we did not send a packet, disable method
debug3: authmethod_lookup password
debug3: remaining preferred: ,password
debug3: authmethod_is_enabled password
debug1: Next authentication method: password
mpiuser@mpiuser1's password:
178024
  • 36
  • 3
  • 15
James Obrien
  • 1
  • 1
  • 1
    Looks like a duplicate of [this one](https://unix.stackexchange.com/questions/247612/ssh-keeps-skipping-my-pubkey-and-asking-for-a-password/247614#247614). – Eduardo Trápani Oct 14 '19 at 20:50
  • try -vvv instead of -v for more verbosity and paste the result here please – Ouss Oct 14 '19 at 21:47
  • can you paste the content of: - client .ssh/authorized_keys - master .ssh/id_rsa.pub .ssh/id_rsa.pub should be in .ssh/authorized_keys – Ouss Oct 14 '19 at 21:54
  • I'd put my money on SELinux. Try temporarily disabling it. If that works, look at audit.log – davidgo Oct 15 '19 at 02:11
  • @EduardoTrápani I have already looked over those responces, his issue was using a deprecated DSA key, im using RSA keys. Nothing in that thread helped me. – James Obrien Oct 15 '19 at 18:59
  • @Ouss I edited my OP with the -vvv flag – James Obrien Oct 15 '19 at 18:59
  • @davidgo I did some research and ubuntu doesnt come stock with SElinux? my machines are all running stock ubuntu 16.04 server – James Obrien Oct 15 '19 at 19:00
  • @EduardoTrápani could you post the output of the ssh server configurations on the host (mpiuser1): $ cat /etc/ssh/sshd_config – Ouss Oct 16 '19 at 09:58

1 Answers1

0

UPDATE - Turns out ssh-copy-id was grabbing some bogus dsa key from god knows where, but manually forcing it to copy my id_rsa.pub key now connecting with ssh via a key works perfectly.

James Obrien
  • 1
  • 1