Lost ssh access to Google Cloud VM

Question

I have a VM (Debian) running on Google Cloud Platform, but I can't connect via ssh or serial console (can't create an user via startup-script for some reason). Already tried a bunch of troubleshooting guides in order to fix it.

I was using the ssh connection previously with no problems at all. The website and databases running on that VM are still working.

I've tried

1 - Checked if firewall entry "default-allow-ssh" exists

2 - Tried connecting with a different user using cmd

gcloud compute ssh another-username@$PROB_INSTANCE

3 - Added metadata "startup-script" key with value:

#! /bin/bash
useradd -G sudo USER
echo 'USER:PASS' | chpasswd

Rebooted (also tried interrupt/start), tried connecting via serial console but it says the login is incorrect. The startup script is not working or not creating my user.

4 - Increased disk size.

5 - Increased memory (upgraded the VM instance type).

6 - Removed ssh keys from both VM details and Metadata tabs, followed by a reboot:

After removing I've tried to generate keys again using command:

gcloud beta compute ssh INSTANCE_NAME -- -vvv

but it returns:

No zone specified. Using zone [us-east1-b] for instance: [INSTANCE_NAME].
Updating project ssh metadata...⠏Updated [https://www.googleapis.com/compute/beta/projects/PROJECT_NAME].
Updating project ssh metadata...done.
Waiting for SSH key to propagate.
USER@IP_ADDRESS: Permission denied (publickey).
USER@IP_ADDRESS: Permission denied (publickey).
USER@IP_ADDRESS: Permission denied (publickey).
USER@IP_ADDRESS: Permission denied (publickey).
USER@IP_ADDRESS: Permission denied (publickey).
USER@IP_ADDRESS: Permission denied (publickey).
USER@IP_ADDRESS: Permission denied (publickey).
USER@IP_ADDRESS: Permission denied (publickey).
USER@IP_ADDRESS: Permission denied (publickey).
USER@IP_ADDRESS: Permission denied (publickey).
USER@IP_ADDRESS: Permission denied (publickey).
USER@IP_ADDRESS: Permission denied (publickey).

More details

Running

gcloud beta compute ssh --zone ZONE INSTANCE_NAME --project PROJECT_NAME

returns:

USER@IP_ADDRESS: Permission denied (publickey).

Running (a second time, after waiting for propagation)

gcloud beta compute ssh INSTANCE_NAME -- -vvv

returns:

[...]
OpenSSH_7.9p1 Debian-10+deb10u2, OpenSSL 1.1.1g  21 Apr 2020
debug1: Reading configuration data /home/USER/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: Applying options for *
debug2: resolve_canonicalize: hostname IP_ADDRESS is address
debug2: ssh_connect_direct
debug1: Connecting to IP_ADDRESS [IP_ADDRESS] port 22.
debug1: Connection established.
debug1: identity file /home/USER/.ssh/google_compute_engine type 0
debug1: identity file /home/USER/.ssh/google_compute_engine-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.9p1 Debian-10+deb10u2
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.4p1 Debian-10+deb9u7
debug1: match: OpenSSH_7.4p1 Debian-10+deb9u7 pat OpenSSH_7.0*,OpenSSH_7.1*,OpenSSH_7.2*,OpenSSH_7.3*,OpenSSH_7.4*,OpenSSH_7.5*,OpenSSH_7.6*,OpenSSH_7.7* compat 0x04000002
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to IP_ADDRESS:22 as 'USER'
debug1: using hostkeyalias: compute.INSTANCE_ID
debug3: hostkeys_foreach: reading file "/home/USER/.ssh/google_compute_known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/USER/.ssh/google_compute_known_hosts:1
debug3: load_hostkeys: loaded 1 keys from compute.INSTANCE_ID
debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-grou
p14-sha256,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519-cert-v01@openssh.com
,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-grou
p14-sha256,diffie-hellman-group14-sha1
debug2: host key algorithms: ssh-rsa,rsa-sha2-512,rsa-sha2-256,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:Or8[...]
debug1: using hostkeyalias: compute.INSTANCE_ID
debug3: hostkeys_foreach: reading file "/home/USER/.ssh/google_compute_known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/USER/.ssh/google_compute_known_hosts:1
debug3: load_hostkeys: loaded 1 keys from compute.INSTANCE_ID
debug1: Host 'compute.INSTANCE_ID' is known and matches the ECDSA host key.
debug1: Found key in /home/USER/.ssh/google_compute_known_hosts:1
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey after 134217728 blocks
debug1: Will attempt key: /home/USER/.ssh/google_compute_engine RSA SHA256:brI3[...] explicit
debug2: pubkey_prepare: done
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred gssapi-keyex,gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /home/USER/.ssh/google_compute_engine RSA SHA256:brI3[...] explicit
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
USER@IP_ADDRESS: Permission denied (publickey).
ERROR: (gcloud.beta.compute.ssh) [/usr/bin/ssh] exited with return code [255].

Update

Followed Alex's suggestions and the serial port output returns:

Welcome to [1mDebian GNU/Linux 9 (stretch)[0m!

[    2.364319] systemd[1]: No hostname configured.
[    2.365157] systemd[1]: Set hostname to <localhost>.
[    3.142016] systemd[1]: google-shutdown-scripts.service: Cannot add dependency job, ignoring: Unit google-shutdown-scripts.service is masked.
[    3.144581] systemd[1]: google-clock-skew-daemon.service: Cannot add dependency job, ignoring: Unit google-clock-skew-daemon.service is masked.
[    3.147589] systemd[1]: google-instance-setup.service: Cannot add dependency job, ignoring: Unit google-instance-setup.service is masked.
[    3.149799] systemd[1]: google-accounts-daemon.service: Cannot add dependency job, ignoring: Unit google-accounts-daemon.service is masked.
[    3.152485] systemd[1]: google-startup-scripts.service: Cannot add dependency job, ignoring: Unit google-startup-scripts.service is masked.

I really hope there is a fix :/

I'd appreciate any help or tips, Thanks!

score 1 · Answer 1 · answered Feb 05 '21 at 16:18

This "Permission denied" error can occur for several reasons.

Did you enable "OS Login"? If OS Login is enabled on your project, your VM doesn't accept SSH keys that are stored in metadata. You have to disable "OS Login"
Or you used an SSH key stored in an OS Login profile to connect to a VM that doesn't have OS Login enabled. So enable "OS Login" and then add your SSH keys to metadata.
If you manually added SSH keys to your VM and then connected to your VM using the Google Cloud Console, Compute Engine created a new key pair for your connection. After the new key pair expired, Compute Engine deleted your ~/.ssh/authorized_keys file in the VM, which included your manually added SSH key. You have to re-add your SSH key to metadata.
If your sshd daemon isn't running or isn't configured properly you can't connect to your VM using SSH. You can connect to your VM via serial port, and then ensure that your sshd_config is set up correctly.

score 0 · Answer 2 · answered Nov 06 '20 at 06:18

0

It is best to check the logs on Serial Console first by using the command

gcloud compute instances get-serial-port-output

Or you can check this link for other ways to do it. From here you should be able to get some answers on why it is failing. You will also be able to see why your startup script for serial connection fails.

You can try this block of code on the startup-script instead:

#! /bin/bash
useradd USERNAME
echo 'USERNAME:PASSWORD!' | chpasswd
usermod -aG wheel USERNAME

For a more detailed troubleshooting steps on SSH on GCE, you can check the GCP official documentation here.

answered Nov 06 '20 at 06:18

Alex G

325
1
7

1

Hey Alex, thanks for your response, but I think that the startup-script is not running at all for some reason... tried your version as well. Regarding the serial port output, that is what I've found: "systemd[1]: google-instance-setup.service: Cannot add dependency job, ignoring: Unit google-instance-setup.service is masked. systemd[1]: google-accounts-daemon.service: Cannot add dependency job, ignoring: Unit google-accounts-daemon.service is masked. systemd[1]: google-startup-scripts.service: Cannot add dependency job, ignoring: Unit google-startup-scripts.service is masked." – krz7 Nov 06 '20 at 07:48
Another approach would be mounting the disk to another VM instance and repairing the SSH configuration from there. This [article](https://serverfault.com/questions/706888/cant-ssh-after-configuring-sshd-config) might help, – Alex G Aug 03 '21 at 22:38

score 0 · Answer 3 · edited Jan 16 '22 at 01:35

SSH:

Use the link above provided by “Alex G" as a reference for troubleshooting ssh. Or find the same link here

Use the command below to check ssh service status

sudo systemctl status sshd

Startup script:

Use the link as a reference to use startup scripts

Use the command below to check installed google systemd services

sudo systemctl list-unit-files | grep google

If services are masked, unmask the services and then enable them.

sudo systemctl unmask "service-name"
sudo systemctl enable "service-name"

Startup script runs when booting up, not in the background after booting up and is not related to SSH. Hence, if the startup script is masked/disabled in the list above, you could try enabling it to automatically start on boot up.

Furthermore, if you have a startup script that needs to run immediately you can use the command below.

sudo systemctl start google-startup-scripts.service.

Use the command below to view output from the startup script.

sudo journalctl -u google-startup-scripts.service

List the google running processes to check for processes needed to SSH. Verify if google-network-daemon.service and google-accounts-daemon.service are running.

sudo ps aux | grep google

If you are still facing issues, reinstall/update the guest environment. Use the link as a reference

Lost ssh access to Google Cloud VM

3 Answers3