1

The problem

After starting an Ubuntu 14.04 EC2 instance with expanded HD volumes, ssh-ing into it fails with Connection refused.

The EBS expansion process

One of my Ubuntu 14.04 EC2 machines was running low on HD size. In order to solve the problem, I followed AWS's own manual on HD expansion:

  • Stopped the machine
  • Detached both volumes
  • Took a snapshot of both volumes
  • Created a larger volumes from the snapshot
  • Attached the new volumes and started the machine

In addition to that, I took the opportunity to add an Elastic IP to the machine, if that matters.

After starting the machine, I constantly get a Connection refused error when ssh-ing to it. I tried ssh from within the VPC to the private IP and from outside. I've used both the XX.X.XXX.XX IP and the ec2-XX-X-XXX-XX.compute-1.amazonaws.com DNS name, and both the original .pem key I've downloaded from AWS upon creation, and the ssh key I've placed in the .ssh/authorized_keys of the machine.

I get the same response:

ssh: connect to host ec2-XX-X-XXX-XX.compute-1.amazonaws.com port 22: Connection refused

Notes / What have I tried

  • I have connected the volumes to another instance and checked them. The files are there.
  • I tried removing from the PermitRootLogin lines from /etc/ssh/sshd_config.
  • I have tried connecting to the machine using the Java client in the EC2 console.
  • I have tried connecting the original volumes to the machine (before expansion). I still get Connection refused.
  • The root EBS volume is connected at /dev/sda1.

Solution

Update: Issue solved. Thanks a bunch to everybody here!

Community
  • 1
Adam Matan
  • 13,194
  • 19
  • 55
  • 75
  • When you mounted the volumes on another instance, did you have a look at the syslog and messages files? What was in them? – Mike Scott Aug 16 '15 at 06:49
  • @MikeScott Great point. Nothing interesting in syslog, but `grep sshd var/log/auth.log` shows `Accepted publickey for ubuntu from port 3858 ssh2: RSA...`! – Adam Matan Aug 16 '15 at 07:00
  • possible duplicate of [What causes the 'Connection Refused' messge?](http://serverfault.com/questions/725262/what-causes-the-connection-refused-messge) – user9517 Sep 28 '15 at 14:11

3 Answers3

1

just putting this dirty hacky script, I used this to debug once (you just need python 2.xx installed on the machine). This is dirty but can help anyway!

Attach and mount your volume to another VM and create a file on it, eg /whatever_mount_path/you_like/cgi-bin/cmd.py ('cgi-bin' is important), with the content as below:

#!/usr/bin/env python
import os
br='<br/>'
print "Content-Type: text/html"
qs=os.environ['QUERY_STRING']
qs = {qs.split('=')[0]:qs.split('=')[1] for qs in   os.environ['QUERY_STRING'].split('&')}
print "cmd: ", qs ['cmd'], br*2
res = os.popen(qs['cmd']).read().replace('\n',br)
print res

then in the "/etc/rc.local" of your volume, add those lines:

cd /path_to_cgi-bin   #should be the path to your cgi-bin directory created above, without the mount path
nohup python -m CGIHTTPServer 8000 >> nohup.out 2>&1 &

reattach your volume to the initial machine, and boot it. From now, you can execute system commands from your web browser, by giving your cmd as GET query string. eg:

http://<YOUR VM IP>:8000/cgi-bin/cmd.py?cmd=netstat -ltpn | grep 22

which will be encoded by your browser as:

http://<YOUR VM IP>:8000/cgi-bin/cmd.py?cmd=netstat%20-ltpn%20|%20grep%2022

just make sure that the port 8000 is free, and that your security group is configured properly

Tom
  • 616
  • 8
  • 13
1

After a chat support session with an AWS supporter, we figured out the problem.

I have expanded two volumes - /dev/sda1 and /dev/sdf. When I connected the expanded volumes to the machine, the second one was attached as a different device for some reason (/dev/sdg, If my memory serves me well). This caused some boot problems and sshd never came up.

When the supporter noted the discrepancy, I've re-attached the volume to /dev/sdf and everything worked well.

Thanks everybody for their help - you've been great!

Adam Matan
  • 13,194
  • 19
  • 55
  • 75
0

Connection refused generally means that there is nothing listening on the relevant IP:Port. Normally you can confirm this by using the OOB console to log in to the system and run netstat, for example

netstat -tunlp | grep :22

would confirm that something was or not listening on port 22.

As you likely don't have an OOB console then you're stuck with mounting the volumes on another instance and digging around in the logs and config files.

Check that your sshd is configured to Listen on the correct interface(es) and Port. While you're there increase the LogLevel so that next time you boot you get more information.

Dig around in all of the logs and see if there are any relevant messages.

I guess you could also add a cron job to the system by simply editing a relevant file and get it to run a netstat command. I don't use Ubuntu but something like

@reboot root netstat -tunlp >>/tmp/netstat.out

in /etc/crontab (or whatever Ubuntu uses) would would get a snapshot of the system after it was booted

Although I expect it would give a different error message ensure that the EC2 security groups are configured correctly.

user9517
  • 115,471
  • 20
  • 215
  • 297