0

I have an Ubuntu instance that I'm running on Azure. I've been using it for the last 6 months or so to level up my novice Linux admin skills. Yesterday I was configuring a Capistrano deployment to this server when my ssh key stopped working all of a sudden. In the past I've seen Permission denied while trying to ssh to the box and I've just readded my private key to the key agent and it worked again. I was never really sure why the key stopped working but since I could fix it without too much hassle and it didn't occur very often I didn't think much of it. Yesterday, though, I wasn't able to fix the problem that way and I've been banging my head trying to figure out the problem since.

Here are the steps I've taken and the things I've tried:

  • Here is what I normally run (which has worked for months up until yesterday): ssh azure-southerneer

  • The relevant entry in my ~/.ssh/config:

    Host azure-southerneer
Hostname southerneer.cloudapp.net
User azureuser
IdentityFile ~/.ssh/azure-wp-eric.key
IdentitiesOnly yes
ServerAliveInterval 300 
StrictHostKeyChecking no
  • The output: Permission denied (publickey).

  • If I run it with the -v flag here is what I get:

    OpenSSH_6.2p2, OSSLShim 0.9.8r 8 Dec 2011
OpenSSH_6.2p2, OSSLShim 0.9.8r 8 Dec 2011
debug1: Reading configuration data /Users/kirkham/.ssh/config
debug1: /Users/kirkham/.ssh/config line 27: Applying options for azure-southerneer
debug1: Reading configuration data /etc/ssh_config
debug1: /etc/ssh_config line 20: Applying options for *
debug1: /etc/ssh_config line 102: Applying options for *
debug1: Connecting to southerneer.cloudapp.net [191.237.4.63] port 22.
debug1: Connection established.
debug1: identity file /Users/kirkham/.ssh/azure-wp-eric.key type -1
debug1: identity file /Users/kirkham/.ssh/azure-wp-eric.key-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_6.2
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1p1 Ubuntu-8
debug1: match: OpenSSH_6.6.1p1 Ubuntu-8 pat OpenSSH*
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: server->client aes128-ctr hmac-md5-etm@openssh.com none
debug1: kex: client->server aes128-ctr hmac-md5-etm@openssh.com none
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(1024<1024<8192) sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_GROUP
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug1: expecting SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Server host key: RSA 77:6f:e6:84:ca:09:04:83:fd:37:96:3f:d3:2a:77:a6
debug1: Host 'southerneer.cloudapp.net' is known and matches the RSA host key.
debug1: Found key in /Users/kirkham/.ssh/known_hosts:78
debug1: ssh_rsa_verify: signature correct
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: SSH2_MSG_NEWKEYS received
debug1: Roaming not allowed by server
debug1: SSH2_MSG_SERVICE_REQUEST sent
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /Users/kirkham/.ssh/azure-wp-eric.key
debug1: read PEM private key done: type RSA
debug1: Authentications that can continue: publickey
debug1: No more authentication methods to try.
Permission denied (publickey).

WTF?! It's trying the correct key, but it just doesn't like it anymore?!

  • I know that the server is up and running because the websites that I'm hosting there still run great. From the verbose ssh output it's obviously responding, it just doesn't like my azure-wp-eric.key (which hasn't changed).
  • I wasn't purposefully toying with permissions on my azureuser home directory on the server or with the .ssh directory, so that shouldn't have mucked up anything.
  • I've tried just bouncing the server from the Azure management portal, but that didn't solve anything.
  • I did run sudo apt-get update on the server yesterday morning, but I was able to ssh fine all through the day while testing my capistrano deployment so that doesn't seem to have caused the problem.

At this point I'm at my wit's end in terms of things to check. I'm obviously not very well versed at troubleshooting ssh problems and so I'm slowly working my way through articles trying to find what I'm missing. I don't know of any way to methodically figure out what I'm doing wrong which, as a developer, really bothers me. Is there something I'm overlooking? Are there other things to verify that I'm on the right or wrong track? Has anyone else experienced similar issues on Ubuntu or Azure VMs? Please let me know if there are any further details that I could supply to help diagnose the problem. Please help!

Community
  • 1
Southerneer
  • 111
  • 4
  • Is `/Users/kirkham/.ssh/azure-wp-eric.key` encrypted? Does `/Users/kirkham/.ssh/azure-wp-eric.key.pub` exist? Do the two keys match? What are the timestamps on those two files? – kasperd Jan 15 '15 at 20:23
  • @kasperd the .key is not encrypted. Here are the permissions on it: `-rw------- 1 kirkham staff ... azure-wp-eric.key`. I don't have the original .pub file that got added to the server so I don't know if it matches or what the timestamp is. – Southerneer Jan 15 '15 at 20:31
  • Try `stat /Users/kirkham/.ssh/azure-wp-eric.key` to check if the file was changed or modified recently. – kasperd Jan 15 '15 at 20:36
  • Here's that output: `16777220 33651479 -rw------- 1 kirkham staff 0 1679 "Jan 15 14:44:54 2015" "Nov 6 14:36:15 2014" "Nov 6 15:11:33 2014" "Nov 6 14:35:31 2014" 4096 8 0 /Users/kirkham/.ssh/azure-wp-eric.key` – Southerneer Jan 15 '15 at 20:48
  • The stat command I am used to formats output differently. Assuming the time stamp today is the atime, then there is nothing in those suggesting the file has been changed. And there is no reason for the server to behave differently depending on whether the key was in an agent or read from file by the ssh client. I am thinking those times you have been adding a key to your agent, you have probably been adding a different key than the one in that file. – kasperd Jan 15 '15 at 21:07
  • @kasperd Figured it out. See my answer. I'm an idiot. Thanks for your help...I will upvote your comments when I have enough rep. – Southerneer Jan 15 '15 at 22:12

1 Answers1

1

I realize my mistake. Thanks @kasperd for helping me figure it out. I followed Azure's SSH creation instructions to the letter. When I generated my key, I used this command

openssl req -x509 -key ~/.ssh/id_rsa -nodes -days 365 -newkey rsa:2048 -out myCert.pem

which generates a new key that expires after 365 days. It just so happens that yesterday was that 365th day which means my key no longer works. Since that was the only key to the VM I guess I'm screwed? Kinda seems like it. Ugh.

Update: Per @kasperd 's comments, the fact that my server stopped letting me SSH in exactly a year after setting it up may just be a crazy coincidence. Thanks to a conversation via the discussion thread over on the Azure documentation site, I discovered that Azure Linux VMs created via the gallery are loaded with the Microsoft Azure Linux Agent which allows admins to reset SSH through the command line (i.e. Powershell). So while it's a pain for me to switch over to Windows, I was able to follow the steps and reset SSH options on my VM. Unfortunately, I still wasn't able to SSH into my machine. Obviously I did something drastic to my machine that wasn't covered by the reset options, but for users with less drastic cases hopefully they can follow some of these links to help their situation.

Southerneer
  • 111
  • 4
  • According to your link the public key placed on the server in standard SSH format without an expiry date. If that's the case, the server won't enforce the expiry date. I don't know if it is being enforced on the client side. The comment you say you used to generate the cert doesn't seem to make sense as you are specifying both an existing key and a new key. – kasperd Jan 15 '15 at 22:29
  • It does indeed sound like the expiry might be enforced somewhere. If the key pair was generated with `ssh-keygen`, then neither private nor public key will specify any expiry date. I don't know which tool you used to generate the `azure-wp-eric.key` file. In the tools usually used to handle ssh keys, the only expiry time I know of is in `ssh-agent`, when loading a key a lifetime can be specified after which the key will be unloaded, this makes sense to use for encrypted keys. – kasperd Jan 15 '15 at 23:21
  • The fact that re-adding the key to the agent seemed to help, it suggests that maybe the key was being added with a limited lifetime. But there are two parts to this, which don't make sense. If the file is un-encrypted and specified in `.ssh/config`, then it doesn't matter if the key is loaded in the agent or not. And having to re-add the key periodically doesn't agree with a fixed expiry time on the key file. I am still thinking that you have been using two different keys, and knowing exactly how that other key is added to the agent would provide a useful hint as to what is going on. – kasperd Jan 15 '15 at 23:27