0

Here is what I do:

  1. I choose Launch Instance from my ec2 dashboard

  2. I select Ubuntu Server 12.04.3 LTS - 64-bit from the AMI list

  3. I choose t1.micro as my instance type

  4. I don't change anything on step2 (Configure Instance Details)

  5. On step 4, I increase the size of the volume from 8 to 200

  6. I click on Review and Launch and run the instance.

Now I can not ssh to the server although its state is running and Status Checks is 2/2 checks passed. I don't have any problem with doing the same steps except for increasing the volume size part. Any idea why this happens?

AliBZ
  • 4,039
  • 12
  • 45
  • 67
  • Same thing happens with `m1.small` instance – AliBZ Nov 28 '13 at 02:35
  • Hard to explain. If you select the supposedly-running instance, then go to Instance Actions, and then Get System Log, that brings up the log from the system console. Anything there? – Michael - sqlbot Nov 28 '13 at 02:45
  • Strange. So, I didn't try it with 200 GB and I didn't try it today, though I did build one with a 12 GB root volume using the same Ubuntu image, this past weekend. It occurs to me that the image might actually be resizing its own filesystem on startup... which is the only possible way changing the size of the root device could actually give you additional space -- because the AMI has a filesystem on it, and it is initially 8GB in size. Now, I'm curious what might be going on under the hood, here... although this isn't technically a "programming" question, I don't think. – Michael - sqlbot Nov 28 '13 at 02:54
  • Yes... the Ubuntu images [apparently do actually automatically resize themselves to fill the disk](http://alestic.com/2010/02/ec2-resize-running-ebs-root) on first boot, and this could logically take some time. How long have you waited? – Michael - sqlbot Nov 28 '13 at 03:01
  • Like 5 minutes, I thought as the ec2 status check is `2/2 checks passed`, it was ready to go. I will wait more for it to see what happens. – AliBZ Nov 28 '13 at 03:03
  • Wow! you are right, I just had to wait for about 10 minutes or so. Thanks. Could you please post your comment as an answer so I could accept it? – AliBZ Nov 28 '13 at 03:07

1 Answers1

3

New EC2 instances with an EBS root volume are started from bootable hard drive snapshots with some metadata, collectively known as an Amazon Machine Image (AMI).

The question prompted me to realize something: given the fact that we're starting off with an existing hard drive image, with an established filesystem on it, it logically follows that no matter how large of a disk that filesystem is copied onto before being bound to our new machine, the filesystem had already been created, and it wouldn't normally be aware of the extra amount of space that happened to be available on the disk, above and beyond its original size.

And yet, I've selected larger-than-default size values for the root disk with the Ubuntu 12.04 LTS AMI and never given a moment's though to the fact that the amount of space I provisioned is "magically" available, when, logically, it shouldn't be. The filesystem should still be 8 GB after bootup, because it's a copy of a filesystem that was originally 8 GB in size, and all of its internal structures should still indicate this.

The only possible conclusion is that the snapshot we're initially booting must actually contain code to automatically grow its own filesystem to fill the disk that it wakes up to find itself running on.

This... turns out to be true. From an early write-up about EC2/EBS by Eric Hammond, describing how to get a larger root volume:

There’s one step left. We need to resize the file system so that it fills up the entire 100 GB EBS volume. Here’s the magic command for ext3. In my early tests it took 2-3 minutes to run. [Update: For Ubuntu 11.04 and later, this step is performed automatically when the AMI is booted and you don’t need to run it manually.] [emphasis added]

— http://alestic.com/2009/12/ec2-ebs-boot-resize

But, you apparently do need to wait for it, and the bigger the disk, the longer the wait. The instance reachability check sounds like a ping test. If so, it's conceivable that the network stack could be up and responsive but sshd might not yet available to accept connections during the resize operation, which would cause the "connection refused" response -- which is an active refusal by the IP stack because the destination socket isn't listening.

The instance reachability check doesn't so much mean the instance is "ready" as much as it means it is "on its way up, or up, and not on its way down."

Community
  • 1
  • 1
Michael - sqlbot
  • 169,571
  • 25
  • 353
  • 427