1

I was installed the ERPNext from Google Click to Deploy since 6 months ago and using the software, the software working until since last week, I can't access to the system and SSH

When connect with SSH the system shown Connection via Cloud Identity-Aware Proxy Failed with code 4003, is anyone can help me fix this?

Any idea how I can get around this problem?

Jeff
  • 13
  • 2
  • 1
    1) Review the Google Cloud Firewall has port 22 set to allow. 2) Reboot your instance. 3) Connect via the serial port and review the logs for the SSH server. 4) Try connecting with the ssh program with debug enabled `ssh -v -i private-key server-ip-address` 5) Post the results in your question. – John Hanley Feb 13 '21 at 09:20
  • 1
    Many thanks, I've checked GCF has port22, after reboot and review the serial port fond one failures, detail below: Failure: File system check of the root filesystem failed The root filesystem on /dev/sdb1 requires a manual fsck. is anyhow to reconnect the OS? – Jeff Feb 13 '21 at 12:13
  • 1
    Serhii Rohoza posted a good answer on the steps required to repair the root file system. Add one additional item and make sure that the file system is not running out of free space after the file system repair. – John Hanley Feb 13 '21 at 18:04
  • @Jeff if you don't have /etc/fstab entry for /dev/sdb1 then you can try to [detach](https://cloud.google.com/compute/docs/disks/detach-reattach-boot-disk#detach_disk) the sdb1 disk from the VM and reboot the VM as I believe root partition /dev/sda1 is consistent enough to start the machine. If so, repair the sdb1 as describe by - SerhiiRohoza and attach it again to check. – Mahboob Feb 13 '21 at 18:38

1 Answers1

3

To fix this issue you should run fsck command on your boot disk.

To do it you can follow steps below:

  1. stop your VM instance (do not delete it):
gcloud compute instances stop [INSTANCE_NAME]

Before detaching the boot disk from the instance, you must stop the instance. There is no need to unmount the disk.

  1. detach the boot disk from your VM instance (usually DISK_NAME is same as INSTANCE_NAME):
gcloud compute instances detach-disk [INSTANCE_NAME] --disk=[DISK_NAME]
  1. create debug VM instance:
gcloud compute instances create debug-instance
  1. attach your boot disk as a non-boot disk to the debug-instance, but don't mount it:
gcloud compute instances attach-disk debug-instance --disk [DISK_NAME] --device-name debug-disk
  1. connect to the debug-instance:
gcloud compute ssh debug-instance
  1. look up the root partition of the disk, which is identified with the part1 notation with a command ls -l /dev/disk/by-id (in this case /dev/sdb1):
$ ls -l /dev/disk/by-id
total 0
lrwxrwxrwx 1 root root  9 Feb 13 14:56 google-debug-disk -> ../../sdb
lrwxrwxrwx 1 root root 10 Feb 13 14:56 google-debug-disk-part1 -> ../../sdb1
lrwxrwxrwx 1 root root 11 Feb 13 14:56 google-debug-disk-part14 -> ../../sdb14
lrwxrwxrwx 1 root root 11 Feb 13 14:56 google-debug-disk-part15 -> ../../sdb15
lrwxrwxrwx 1 root root  9 Feb 13 14:56 google-persistent-disk-0 -> ../../sda
lrwxrwxrwx 1 root root 10 Feb 13 14:56 google-persistent-disk-0-part1 -> ../../sda1
lrwxrwxrwx 1 root root 11 Feb 13 14:56 google-persistent-disk-0-part14 -> ../../sda14
lrwxrwxrwx 1 root root 11 Feb 13 14:56 google-persistent-disk-0-part15 -> ../../sda15
lrwxrwxrwx 1 root root  9 Feb 13 14:56 scsi-0Google_PersistentDisk_debug-disk -> ../../sdb
lrwxrwxrwx 1 root root 10 Feb 13 14:56 scsi-0Google_PersistentDisk_debug-disk-part1 -> ../../sdb1
lrwxrwxrwx 1 root root 11 Feb 13 14:56 scsi-0Google_PersistentDisk_debug-disk-part14 -> ../../sdb14
lrwxrwxrwx 1 root root 11 Feb 13 14:56 scsi-0Google_PersistentDisk_debug-disk-part15 -> ../../sdb15
lrwxrwxrwx 1 root root  9 Feb 13 14:56 scsi-0Google_PersistentDisk_persistent-disk-0 -> ../../sda
lrwxrwxrwx 1 root root 10 Feb 13 14:56 scsi-0Google_PersistentDisk_persistent-disk-0-part1 -> ../../sda1
lrwxrwxrwx 1 root root 11 Feb 13 14:56 scsi-0Google_PersistentDisk_persistent-disk-0-part14 -> ../../sda14
lrwxrwxrwx 1 root root 11 Feb 13 14:56 scsi-0Google_PersistentDisk_persistent-disk-0-part15 -> ../../sda15
  1. run a file system check on the root partition:
sudo fsck /dev/sdb1

for example:

debug-instance:~$ sudo fsck /dev/sdb1
fsck from util-linux 2.33.1
e2fsck 1.44.5 (15-Dec-2018)
/dev/sdb1: clean, 53782/647168 files, 396250/2588667 blocks
  1. stop the debug-instance VM instance and detach your boot disk from it:
gcloud compute instances stop debug-instance
gcloud compute instances detach-disk debug-instance --disk [DISK_NAME]
  1. reattach the boot disk to your VM instance:
gcloud compute instances attach-disk [INSTANCE_NAME] --disk=[DISK_NAME] --boot
  1. start your VM instance:
gcloud compute instances start [INSTANCE_NAME]
  1. check boot log of your VM instance via serial port.
  2. delete debug-instance:
gcloud compute instances delete debug-instance

In addition, have a look a the documentation Detaching and Reattaching Boot Disks and Troubleshooting VM start up.

Serhii Rohoza
  • 4,287
  • 2
  • 16
  • 29