1

I'd like to solve the following problem using command line:

I'm trying to run the following PoC script from a GCE VM in project-a.

gcloud config set project project-b
gcloud compute instances create gce-vm-b --zone=us-west1-a
gcloud compute ssh --zone=us-west1-a gce-vm-b -- hostname

The VM is created successfully:

NAME ZONE MACHINE_TYPE PREEMPTIBLE INTERNAL_IP EXTERNAL_IP STATUS 
gce-vm-b us-west1-a n1-standard-16 10.12.34.56 12.34.56.78 RUNNING

But get the following error when trying to SSH:

WARNING: The public SSH key file for gcloud does not exist. 
WARNING: The private SSH key file for gcloud does not exist. 
WARNING: You do not have an SSH key for gcloud. 
WARNING: SSH keygen will be executed to generate a key. 
Generating public/private rsa key pair. 
Your identification has been saved in /root/.ssh/google_compute_engine. 
Your public key has been saved in /root/.ssh/google_compute_engine.pub. 
The key fingerprint is: 
...
Updating project ssh metadata... 
.....................Updated [https://www.googleapis.com/compute/v1/projects/project-b]. 
>.done. 
>Waiting for SSH key to propagate. 
>ssh: connect to host 12.34.56.78 port 22: Connection timed out 
>ERROR: (gcloud.compute.ssh) Could not SSH into the instance. It is possible that your SSH key has not propagated to the instance yet. Try running this command again. If you still cannot connect, verify that the firewall and instance are set to accept ssh traffic.

Running gcloud compute config-ssh hasn't changed anything in the error message. It's still ssh: connect to host 12.34.56.78 port 22: Connection timed out

I've tried adding a firewall rule to the project:

gcloud compute firewall-rules create default-allow-ssh --allow tcp:22

.

Creating firewall... 
...........Created [https://www.googleapis.com/compute/v1/projects/project-b/global/firewalls/default-allow-ssh]. 
done. 
NAME NETWORK DIRECTION PRIORITY ALLOW DENY 
default-allow-ssh default INGRESS 1000 tcp:22

The error is now Permission denied (publickey).

gcloud compute ssh --zone=us-west1-a gce-vm-b -- hostname

.

Pseudo-terminal will not be allocated because stdin is not a terminal. 
Warning: Permanently added 'compute.4123124124324242' (ECDSA) to the list of known hosts. 
Permission denied (publickey). 
ERROR: (gcloud.compute.ssh) [/usr/bin/ssh] exited with return code [255].

P.S. The project-a "VM" is a container run by Prow cluster (which is run by GKE).

Ark-kun
  • 6,358
  • 2
  • 34
  • 70

1 Answers1

1

"Permission denied (publickey)" means it is unable to validate the public key for the username.

You haven't specified the user in your command, so the user from the environment is selected and it may not be allowed into the instance gce-vm-b. Specify a valid user for the instance in your command according to the public SSH key metadata.

Héctor Neri
  • 1,384
  • 9
  • 13
  • user@gce-vm-a:~$ gcloud compute ssh --zone=us-west1-a user@gce-vm-b --project=project --verbosity=debug --ssh-flag="-vvv" -- hostname – Ark-kun Sep 06 '18 at 22:41
  • Wow. Adding my-user@instance fixed the problem, but I do not understand it >_< I have many more questions now: 1) How can some VM I never logged into in a project-b I cannot access "impersonate" me - It's only giving my user name to the project-a. Why does it get access now? 2) The gce-vm-a runs as root. I wonder how to make it so I do not need to specify the user in the ssh command. This looks messy, due to the next question: – Ark-kun Sep 06 '18 at 23:13
  • 3) I found out that when I run `gcloud compute ssh nonexistent-user@gce-vm-b`, the key for that user is added to the project-b SSH Keys. If I specify "root", the command fails though. In project-b metadata I see several keys for "root", at least one of which clearly originating from my machine. I do not understand how I created that root entry and why I cannot ssh root@gce-vm-b even from my workstation - usually it can SSH as any user and in this case that user (root) even exists in the project metadata. – Ark-kun Sep 06 '18 at 23:16
  • I also wonder who created other root SSH Key entries. I suspect that the gce-vm-a did that while trying to ssh to gce-vm-b. It's strange that it has succeeded adding the keys for root, but not succeeding in connecting as root. – Ark-kun Sep 06 '18 at 23:16
  • 1) Do you mean that your user "my-user" does not have access to project-b? Maybe it's only creating the user within the VM, not within your project, see the answer for 3). 2) In order to use another account, you have to run "gcloud auth login " and then "gcloud config set account ". This will set your gcloud to use a different user (note that you may see root@project-b, but you can get your account by using "gcloud config list"). – Héctor Neri Sep 07 '18 at 20:04
  • 3) It seems that when you run the command from an authorized user, the "nonexistent-user" is created within the VM, as you would see by running "cut -d: -f1 /etc/passwd". In the case of root, this is an existing user by default and it fails. To SSH as root, check [these answers from this related question](https://stackoverflow.com/questions/26835235/google-cloud-compute-vm-instances). – Héctor Neri Sep 07 '18 at 20:04
  • Thank you for the insights. I think I mostly found out what's happening: gcloud ssh adds arbitrary SSH keys to target project. These SSH keys are provisioned to the instances like gce-vm-b and allow loging with that username. gca-vm-a is a Kubernetes container. By default they're run as root. This user already exists on gce-vm-b, so the key provisioning fails (even though the key is added to the project). Some existing accounts like news exhibit the same behavior. Accounts added via useradd pose no problem though - you can gcloud ssh to them without knowing the password (somewhat strange). – Ark-kun Sep 07 '18 at 21:23
  • One other problem I had was the pollution of the project metadata. Every time new container created a VM, an SSH key was added to project-b metadata. I tried to use instance-level metadata, but nothing helped. (One problem was that gcloud docs use square brackets incorrectly in some places - sometimes they do not indicate optional parameters.) What I found out is that gcloud ssh always adds the keys to the project-level metadata except in some specific situations (instance blocks project-level SSH keys or instance uses legacy-style `sshKeys` instead of `ssh-keys` etc). – Ark-kun Sep 07 '18 at 21:28
  • Indeed, Compute Engine can automatically generate these keys for you [when you Connect to instances](https://cloud.google.com/compute/docs/instances/connecting-to-instance#gcetools). Although you could do all [key management manually](https://cloud.google.com/compute/docs/instances/adding-removing-ssh-keys) the documentation recommends to use [OSLogin](https://cloud.google.com/compute/docs/instances/managing-instance-access) in order to avoid the related risks. – Héctor Neri Sep 10 '18 at 19:39