0

I have a google VM instance which was running flawlessly. It has Centos-7 and Plesk installed in it.

I just stopped it, upgraded machine type (to better CPU and RAM) and started it again. My server stopped responding at all. No websites are running, I can't connect to SSH & Google Cloud SDK Shell is unable to reach server. It says NETWORK ERROR, CONNECTION TIMED OUT. My all other instances works well.

I tried rebooting & resetting multiple times. Reading out stuff from internet since last 6 hours but for no luck. I also tried to clone disk of the instance and creating new instance with the cloned disk but for no luck. Same network connection issue. May be something in OS got corrupted? Please suggest. I have a number of websites hosted on the server which are down due to this. Thanks a lot in advance.

Kunal Khanna
  • 121
  • 2
  • 10
  • Have a look at this [article](https://medium.com/google-cloud/resolving-getting-locked-out-of-a-compute-engine-85800251890b). Then create user/password, enable serial console, restart your VM instance and add full boot log to your question using [pastebin](https://pastebin.com). – Serhii Rohoza Jul 30 '20 at 19:13
  • Being connected via serial console check other logs, interfaces and so on. – Serhii Rohoza Jul 30 '20 at 19:14
  • Now I've found the issue. It is a glitch by google. I now need to connect my hard disk to some other running instance and downgrade some packages. Could anyone of you please help. I can pay for your kind service and valuable time. – Kunal Khanna Jul 30 '20 at 19:54
  • It's known issue. You can follow it at the [Google Issue Tracker case](https://issuetracker.google.com/issues/162523000) or at the [Google Cloud Status Dashboard](https://status.cloud.google.com/incident/compute/20009). – Serhii Rohoza Jul 31 '20 at 09:22

1 Answers1

1

I took screenshot of my VM using Google Cloud Shell. It is as follows:

VM Instance Screenshot

I connected with serial console which is as follows:

enter image description here

While creating a ticket with google, I found that they've posted some information under "Known Issues". I am pasting the whole stuff as there is no direct link to reach there. The symptoms they told were exactly what was happening to me:

Below is Known Issue Posted by Google:

Description: We are experiencing an issue with Google Compute Engine instances running RHEL and CentOS 7 and 8. More details on this issue are available in the following article and bugs: https://access.redhat.com/solutions/5272311 https://bugzilla.redhat.com/show_bug.cgi?id=1861977 (RHEL 8) https://bugzilla.redhat.com/show_bug.cgi?id=1862045 (RHEL 7) Symptoms: Instances running RHEL and CentOS 7 and 8 that run yum update may fail to boot after restart with errors messages referring to a combination of: "X64 Exception Type - 0D(#GP - General Protection) CPU Apic ID", "FXSAVE_STATE" or "Find image based on IP". This issue affects instances with specific versions of the shim package installed. To find the currently installed shim version, use the following command: rpm -q shim-x64 Affected shim versions: CentOS 7: shim-x64-15-7.el7_9.x86_64 CentOS 8: shim-x64-15-13.el8.x86_64 RHEL 7: shim-x64-15-7.el7_8.x86_64 RHEL 8: shim-x64-15-14.el8_2.x86_64 Workaround: Do not update or reboot instances running RHEL or CentOS 7 and 8. If you are on an affected shim version, run yum downgrade shim\* grub2\* mokutil to downgrade to the correct version. This command may not work on CentOS 8. If you have already rebooted, you will need to attach the disk to another instance, chroot into the disk, then run the yum downgrade command. We will provide an update by Thursday, 2020-07-30 14:00 US/Pacific with current details.

Start time: July 30, 2020 at 9:08:34 PM GMT+5

How to diagnose: Instances running RHEL and CentOS 7 and 8 that run yum update may fail to boot after restart with errors messages referring to a combination of: "X64 Exception Type - 0D(#GP - General Protection) CPU Apic ID", "FXSAVE_STATE" or "Find image based on IP". This issue affects instances with specific versions of the shim package installed. To find the currently installed shim version, use the following command: rpm -q shim-x64 Affected shim versions: CentOS 7: shim-x64-15-7.el7_9.x86_64 CentOS 8: shim-x64-15-13.el8.x86_64 RHEL 7: shim-x64-15-7.el7_8.x86_64 RHEL 8: shim-x64-15-14.el8_2.x86_64

Workaround: Do not update or reboot instances running RHEL or CentOS 7 and 8. If you are on an affected shim version, run yum downgrade shim\* grub2\* mokutil to downgrade to the correct version. This command may not work on CentOS 8. If you have already rebooted, you will need to attach the disk to another instance, chroot into the disk, then run the yum downgrade command.

Kunal Khanna
  • 121
  • 2
  • 10