1

We have a dev server running jenkins with various jobs that run shell scripts that connect to our live server via SSH (using key-based authentication).

This doesn't seem to work reliably.

If I test the SSH connection on its own by logging in to the dev server as the "jenkins" user and then SSH'ing to the live server, the authentication works and logs me in without the password prompt.

If I run one of the jenkins jobs manually it works most of the time (although one failed this morning prompting me to write this ticket) but if I automate the job to run nightly it fails most of the time (I think it has worked once).

The point of failure each time is at the SSH connection:

> ssh -T nepton@xxx
> Permission denied, please try again.
> Permission denied, please try again.
> Permission denied (publickey,password).

Any help would be greatly appreciated as I'm completely stuck and Linux is definitely not my area of expertise.

Additional Information:

  • I generated the public key on Dev (whilst logged in as "jenkins" user) using: ssh-keygen -t rsa
  • I copied the key to the live server using: ssh-copy-id -i ~/.ssh/id_rsa.pub nepton@xxx

Permissions of /var/lib/jenkins/.ssh/ on Dev are:

drwx------  2 jenkins jenkins 4096 Apr  4 13:31 .
drwxr-xr-x 13 jenkins jenkins 4096 Apr  4 13:38 ..
-rw-------  1 jenkins jenkins 1679 Apr  4 13:31 id_rsa
-rw-r--r--  1 jenkins jenkins  400 Apr  4 13:31 id_rsa.pub
-rw-r--r--  1 jenkins jenkins 1338 Apr  4 13:33 known_hosts

Permissions of /home/nepton/.ssh/ on the Live server are:

drwx------ 2 nepton nepton 4096 Mar 20 17:37 .
drwx------ 4 nepton nepton 4096 Apr  2 10:52 ..
-rw------- 1 nepton nepton 2005 Apr  8 10:23 authorized_keys
-rw-r--r-- 1 nepton nepton 1106 Apr  1 10:39 known_hosts
  • I'm running the latest LTS version of Ubuntu and the standard SSH package on both servers.

Debug information

ssh -v -T -i /var/lib/jenkins/.ssh/id_rsa nepton@xxx
OpenSSH_5.9p1 Debian-5ubuntu1, OpenSSL 1.0.1 14 Mar 2012
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug1: Connecting to xxx [xx.xxx.xx.xxx] port 22.
debug1: Connection established.
debug1: identity file /var/lib/jenkins/.ssh/id_rsa type 1
debug1: Checking blacklist file /usr/share/ssh/blacklist.RSA-2048
debug1: Checking blacklist file /etc/ssh/blacklist.RSA-2048
debug1: identity file /var/lib/jenkins/.ssh/id_rsa-cert type -1
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.9p1 Debian-5ubuntu1
debug1: match: OpenSSH_5.9p1 Debian-5ubuntu1 pat OpenSSH*
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_5.9p1 Debian-5ubuntu1
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5 none
debug1: kex: client->server aes128-ctr hmac-md5 none
debug1: sending SSH2_MSG_KEX_ECDH_INIT
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ECDSA xxx
debug1: Host 'xxx' is known and matches the ECDSA host key.
debug1: Found key in /var/lib/jenkins/.ssh/known_hosts:7
debug1: ssh_ecdsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /var/lib/jenkins/.ssh/id_rsa
debug1: Authentications that can continue: publickey,password
debug1: Next authentication method: password
debug1: read_passphrase: can't open /dev/tty: No such device or address
debug1: Authentications that can continue: publickey,password
Permission denied, please try again.
debug1: read_passphrase: can't open /dev/tty: No such device or address
debug1: Authentications that can continue: publickey,password
Permission denied, please try again.
debug1: read_passphrase: can't open /dev/tty: No such device or address
debug1: Authentications that can continue: publickey,password
debug1: No more authentication methods to try.
Permission denied (publickey,password).
Build step 'Execute shell' marked build as failure
MeatPopsicle
  • 133
  • 1
  • 5
  • Could you check your server logs if you find something. DNS issue ? – Dom Apr 08 '13 at 09:13
  • Edit the scripts to use `ssh -v -T nepton@xxx`. That will give you a lot more information; add the output to your question. – Jenny D Apr 08 '13 at 09:16
  • Make sure you make the first connection logged in as jenkins, that way you can confirm the acceptance of the key. – NickW Apr 08 '13 at 09:18
  • @Dom I had a look in /var/log/syslog and /var/log/jenkins and found nothing useful – MeatPopsicle Apr 08 '13 at 09:58
  • @JennyD Good idea, thanks. The automated script runs overnight so I will post the information tomorrow. – MeatPopsicle Apr 08 '13 at 10:00
  • @NickW The connection works logged in as jenkins. I am wondering if the scheduled jenkins job uses the same user though, I'll add an "env" command to the script to see what it says. – MeatPopsicle Apr 08 '13 at 10:01
  • 1
    Yeah, I've tripped over the accept server key (known hosts file) a few times in the past. – NickW Apr 08 '13 at 10:05
  • @JennyD I've added the -v you suggested and put the dump in the original post. I think it's looking in the wrong place for the private key "Trying private key: /var/lib/jenkins/.ssh/id_dsa" – MeatPopsicle Apr 09 '13 at 07:21
  • Also I can no longer type "ssh nepton@xxx" and connect to the Live server without the password, it's like it forgets the key overnight or something – MeatPopsicle Apr 09 '13 at 07:29

2 Answers2

1

As you say in comments, it's not using the key you want it to. You can point out the correct key like this:

ssh -T -i /path/to/real/id_rsa nepton@xxx
user9517
  • 115,471
  • 20
  • 215
  • 297
Jenny D
  • 27,780
  • 21
  • 75
  • 114
  • This worked the day I tried it but no longer works today. I have updated the debug information in my original post. Thanks. – MeatPopsicle Apr 11 '13 at 06:53
1

I had this same issue and found a solution that may be related here. If the system you are attempting to SSH to is behind a load balancer (or something similar) it's possible that sometimes your SSH can succeed, and sometimes it will fail if the machines that are being load balanced do not have a shared home directory (or at least your .ssh information such as the public key).

From your side, it will look like you are logging into a single host, but actually it is intermittently a different actual host, which could explain this odd behavior.

jrel
  • 11
  • 1