can't ssh to download data from gcloud instance running centos6 to get moodle backup due to possible hack

Question

I have a centos 6 lamp moodle server in a gcloud vm instance, 2 days ago users reported issues accessing moodle with a database connection error.

Through my research this is what I found:

restarting the instance didn't work and it actually cause an issue with apache as I can't even server html pages from the server eventhough the httpd service is up and restarts ok.
checked apache and security logs, the logs seem to indicate an attempt of hacking, and this is my fault and possible the reason why this is happenning.

I figure I would try to get my data and start over on a new instance but when I try to ssh into the server (ssh web or gcloud ssh cli or filezilla) to download a backup zip file that gets created automatically via a script I setup, I can't seem to be able to ssh at all here are some details:

on browser ssh I get this error:

Error: The VM guest environment is outdated and only supports the deprecated 'sshKeys' metadata item. Please follow the steps here to update

and even though I'm eventually given access I can't download anything via browser.
on ssh gcloud cli I get this error:

Permission denied (publickey,gssapi-keyex,gssapi-with-mic). ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255].

ssh verbose shows this:

MacBook-Pro:.ssh fcortes$ sudo gcloud compute ssh centos6 --ssh-flag="-vvv"
Password:
OpenSSH_7.9p1, LibreSSL 2.7.3
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 48: Applying options for *
debug2: resolve_canonicalize: hostname 34.66.109.207 is address
debug2: ssh_connect_direct
debug1: Connecting to 34.66.109.207 [34.66.109.207] port 22.
debug1: Connection established.
debug1: identity file /Users/fcortes/.ssh/google_compute_engine type 0
debug1: identity file /Users/fcortes/.ssh/google_compute_engine-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_7.9
debug1: Remote protocol version 2.0, remote software version OpenSSH_5.3
debug1: match: OpenSSH_5.3 pat OpenSSH_5* compat 0x0c000002
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to 34.66.109.207:22 as 'root'
debug1: using hostkeyalias: compute.8844278207473884047
debug3: hostkeys_foreach: reading file "/Users/fcortes/.ssh/google_compute_known_hosts"
debug3: record_hostkey: found key type RSA in file /Users/fcortes/.ssh/google_compute_known_hosts:1
debug3: load_hostkeys: loaded 1 keys from compute.8844278207473884047
debug3: order_hostkeyalgs: prefer hostkeyalgs: rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1,ext-info-c
debug2: host key algorithms: rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: diffie-hellman-group-exchange-sha256,diffie-hellman-group-exchange-sha1,diffie-hellman-group14-sha1,diffie-hellman-group1-sha1
debug2: host key algorithms: ssh-rsa,ssh-dss
debug2: ciphers ctos: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: ciphers stoc: aes128-ctr,aes192-ctr,aes256-ctr,arcfour256,arcfour128,aes128-cbc,3des-cbc,blowfish-cbc,cast128-cbc,aes192-cbc,aes256-cbc,arcfour,rijndael-cbc@lysator.liu.se
debug2: MACs ctos: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: MACs stoc: hmac-md5,hmac-sha1,umac-64@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-ripemd160,hmac-ripemd160@openssh.com,hmac-sha1-96,hmac-md5-96
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos: 
debug2: languages stoc: 
debug2: first_kex_follows 0 
debug2: reserved 0 
debug1: kex: algorithm: diffie-hellman-group-exchange-sha256
debug1: kex: host key algorithm: ssh-rsa
debug1: kex: server->client cipher: aes128-ctr MAC: umac-64@openssh.com compression: none
debug1: kex: client->server cipher: aes128-ctr MAC: umac-64@openssh.com compression: none
debug3: send packet: type 34
debug1: SSH2_MSG_KEX_DH_GEX_REQUEST(2048<3072<8192) sent
debug3: receive packet: type 31
debug1: got SSH2_MSG_KEX_DH_GEX_GROUP
debug2: bits set: 1534/3072
debug3: send packet: type 32
debug1: SSH2_MSG_KEX_DH_GEX_INIT sent
debug3: receive packet: type 33
debug1: got SSH2_MSG_KEX_DH_GEX_REPLY
debug1: Server host key: ssh-rsa SHA256:FtoD8qoSAeNtVsWukZ5YTbWIiApxpn2T+VsVzcF7SPE
debug1: using hostkeyalias: compute.8844278207473884047
debug3: hostkeys_foreach: reading file "/Users/fcortes/.ssh/google_compute_known_hosts"
debug3: record_hostkey: found key type RSA in file /Users/fcortes/.ssh/google_compute_known_hosts:1
debug3: load_hostkeys: loaded 1 keys from compute.8844278207473884047
debug1: Host 'compute.8844278207473884047' is known and matches the RSA host key.
debug1: Found key in /Users/fcortes/.ssh/google_compute_known_hosts:1
debug2: bits set: 1531/3072
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey after 4294967296 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey after 4294967296 blocks
debug1: Will attempt key: /Users/fcortes/.ssh/google_compute_engine RSA SHA256:CxsVOSJsVd6lZxknFbIRRlzARrtvXpyOItST5IG6mVw explicit
debug2: pubkey_prepare: done
debug3: send packet: type 5
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug3: start over, passed a different list publickey,gssapi-keyex,gssapi-with-mic
debug3: preferred publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Offering public key: /Users/fcortes/.ssh/google_compute_engine RSA SHA256:CxsVOSJsVd6lZxknFbIRRlzARrtvXpyOItST5IG6mVw explicit
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,gssapi-keyex,gssapi-with-mic
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
root@34.66.109.207: Permission denied (publickey,gssapi-keyex,gssapi-with-mic).

and in my console log the only thing that I found that might be pertinent to my problem is this:

Starting udev: udevd[351]: can not read '/etc/udev/rules.d/75-persistent-net-generator.rules'

udevd[351]: can not read '/etc/udev/rules.d/75-persistent-net-generator.rules'

but gcloud console does show that I have my rules setup for http and ssh

https://prnt.sc/o241br

score 0 · Answer 1 · answered Jun 16 '19 at 18:10

Since data is what's important here and as per suggested by Rick Swanson over at facebook group: Linux (RedHat/CentOS), rsync was the solution to get my data/backup

I do have a test server at digital ocean and after doing:

rysnc -avz [backup_path_on_gcloud]/backups digitalocean_user@[digital_ocean_server_ip]

my backups in gcloud were magically transferred over to my test server at digital ocean now onto restoring the backup

score 0 · Answer 2 · answered Jun 17 '19 at 13:34

If you are having issues with connecting to your VM instance in GCP you can try one of the following:

Troubleshooting SSH. You can check if one of scenarios explained in this doc is applicable to your case.
Try connecting to a serial console. You should enable it serial console, add your public SSH key to the project or instance metadata, and store your private key on the local machine from which you want to connect.
If there is still no way to connect, you can take the disk used by VM instance and attach it to another VM as a secondary disk. (skip the "format the disk" from this doc as it will delete all data).

Thanks for letting me know if one of these steps was successful.

can't ssh to download data from gcloud instance running centos6 to get moodle backup due to possible hack

2 Answers2