0

It turns out that our GCE disk is full, which is odd because when I create a snapshot it's only 16gb while the disk is 50gb.

I've tried resizing the disk, that doesn't work. Same problem.

I've tried turning on the serial console but when I connect mysql continually tries to start and never puts me into a login prompt. If I hit enter to get to a prompt, it says that my credentials are incorrect because mysql is still spooling.

I've tried attaching the disk to another engine as a second disk and mysql STILL tries to start even though it's not the boot disk.

This seems like an issue that google needs to address. Their resizing feature does not seem to work and mounting the disk on another server does not work. I cannot resize the disk and cannot mount it to resize on another machine because it too will not start.

Any guidance from others who've experienced this situation would be helpful. This is our production server and we've been down for more than 30 hours!

Thanks.

rvs
  • 4,125
  • 1
  • 27
  • 31
cyberthreat
  • 1
  • 1
  • just to be clear, are you running this on Windows or a Linux instance? – Patrick W Aug 22 '18 at 16:11
  • ubuntu linux 16.4 – cyberthreat Aug 23 '18 at 01:57
  • Keep in mind that resizing a disk partition requires some disk space to update the partition table. If your disk has reached 100% capacity, you can no longer write new entries to the disk at all which means making any changes at all becomes impossible – Patrick W Aug 23 '18 at 14:05
  • 1
    You SHOULD be able to take a [snapshot](https://cloud.google.com/compute/docs/disks/create-snapshots) of the disk, create a new disk from the snapshot and mount the disk to another linux instance. The MySQL service might get stuck in spooling, but as long as it is not the boot disk, you should still be able to interact with the instance OS to make changes to the MySQL disk (including stopping the MySQL application and resizing the disk) – Patrick W Aug 23 '18 at 14:07
  • 1
    As a side note, you should never use your boot disk to store data, instead, always add a secondary disk to use for storage (especially databases). The cost will come to the same. – Patrick W Aug 23 '18 at 14:08

1 Answers1

0

Most likely you've resized a disk (as in, block device), but not a partition and/or a filesystem.

Consider using something like parted to resize partitions and filesystem. If you add output of fdisk -l for the disk more specific guidance can be provided.

See also:

rvs
  • 4,125
  • 1
  • 27
  • 31
  • I am unable to run any commands on the machine. I cannot login via ssh and startup scripts don't run because the disk is full. I get an error from the GCE as such. I am unable to login into the machine because mysql keeps spooling and it keeps telling me that i have bad credentials. – cyberthreat Aug 23 '18 at 13:56
  • I don't follow. Full disk would not prevent you from logging into machine. Nor broken mysql, unless you use some interesting authentication method. – rvs Aug 23 '18 at 18:11
  • the process of establishing an SSH connection still needs to write to the disk. If the disk is full, you won't be able to SSH either – Patrick W Aug 23 '18 at 18:56
  • @PatrickW ah - I think I know what you are talking about. By default openssh does not require any disk writes to succeed to allow log in. However `gcloud ssh` does require temporary ssh key to be written. @John, do you have a user that has non-temporary key, or a password authenication enabled? If not, try the above advice about attaching disk to another instance. – rvs Aug 24 '18 at 13:39