How do you securely retrieve the ssh host keys from a google compute engine instance?

Question

I want to update my ~/.ssh/known_hosts with the host key information for a newly created GCE instance. But I'm not sure how to securely retrieve that information.

I thought something like

gcloud compute ssh <GCEUSER>@<GCEHOST> --command='ssh-keyscan 127.0.0.1'

might work. But that (per the gcloud compute ssh documentation) appears to just be a wrapper for ssh (and, based on seeing StrictHostKeyChecking=no in the parameters listed in the associated log file under $HOME/.config/gcloud/logs/, apparently isn't doing any sort of checking on the host's identity).

There does seem to be a way to use the web console to launch a browser-based ssh session (and interactively/manually run ssh-keyscan), but 1) I can't see the internals to know if it really is as secure as it should be and 2) isn't an effective API for script integration.

Is there an API/gcloud mechanism for securely retrieving the GCE instance's host key?

Answer 1

OP linked to bug 35907612 (initially mentioned by Rahi) which was marked “fixed” with a link to the documentation Storing host keys.

To get the host key, you need to enable “guest attributes” before or during creation of the GCE instance. I enabled it project-wide (not sure why it was disabled by default). Alternatively, you have a chance to enable it when you create the instance.

According to the documentation, gcloud is required to populate the guest attributes, but I seem to be able to jump to step “Confirming that host keys are stored as guest attributes” right after the instance boots up.

notes

1. In Step 2 (confirming keys), I chose Option 2, because I didn’t ask gcloud to populate ~/.ssh/google_compute_known_hosts in last step, so I want to see the keys, not just verifying that they are “somewhere”.

2. When you run get-guest-attributes sub-command, both --project and --zone flags are optional, and they default to the configuration set with gcloud config.

3. To get the SHA-256 hash for host keys directly from get-guest-attributes, with help of jq:

   gcloud compute instances get-guest-attributes my-instance --format json | jq -r '.[] | "\(.key) \(.value) my-server-name"' | ssh-keygen -l -f -
   

   Explain: -r means --raw-output in jq. -l means listing keys with hashes. -f - means file from stdin. my-server-name is optional comment that appears in output (otherwise you'll see no comment).

Answer 2

"gcloud command" or "API" will only retrieve information related to the specific project or Google products, such an instance metadata from a GCE instance. It is not possible to retrieve instance's configuration such as host keys via the gcloud command or API.

Update

SSH host key pair, A private and public key are kept on the SSH server. The public key is shared with the SSH client on each connection. This is used to authenticate the SSH server to the client. After the first connection, the “known host” key is typically stored in ~/.ssh/known_hosts on the SSH client (your computer, for example), though gcloud stores it in a different place.

When you use gcloud compute ssh, your SSH client behaves slightly differently: the server’s public host key is automatically accepted for you the first time you connect, and it’s written to ~/.ssh/google_compute_known_hosts instead.

Every time you connect to a SSH server, it sends its public host key. In most circumstances, the server’s public host key won’t change. After you’ve connected once and saved the public key from your first connection, your SSH client can check to see if the server’s public host key is different. If different, you’ll see a message saying:

Host key verification failed.

When you see this message, one of the following is the case:

• You’re connecting to a different server. This isn’t always malicious. For example, if you replace a server with a new one, you’ll see this message.
• You’re connecting to the same server, but its SSH host keys have been re-generated. Again, this isn’t automatically an indicator of malicious behavior. For example, an administrator could have used ssh-keygen to create a new host key pair.

By default, gcloud compute ssh does not perform strict host key checking. There are reasons for this; however, you can enable strict host key checking by adding the --oStrictHostKeyChecking=yes option to gcloud compute ssh. Here’s an example:

gcloud compute ssh [INSTANCE_NAME] --zone [ZONE] -- -oStrictHostKeyChecking=yes

This is a good example of how you can add arbitrary SSH arguments to gcloud compute ssh.

Skipping host key checking isn’t a problem. Suppose you have created a new instance to which you’ve never connected. How would you know that you’re connecting to that instance and not somewhere else?

Consider the following example. Suppose you have no SSH user key pair for gcloud to use. (If you move both ~/.ssh/google_compute_engine and ~/.ssh/google_compute_engine.pub aside, you can test this.) Suppose you’re not logged into gcloud. You can log out with gcloud auth revoke.

Now try to SSH into an instance using gcloud compute ssh. You’ll have to authenticate to gcloud first. So, run gcloud auth login and sign in. Now your system has an access token stored in ~/.config/gcloud/. This token is used for authentication each time you make an API call using gcloud.

Now try to SSH into the same instance using gcloud compute ssh again. Without a SSH user key pair, it will call ssh-keygen on your behalf to create a new SSH user key pair. The private (user) key will be stored in ~/.ssh/google_compute_engine, and the public (user) key will be stored in ~/.ssh/google_compute_engine.pub. The private key remains local to your computer, and you’re responsible for securing it. The contents of the public key are sent via an API call to projects.setCommonInstanceMetadata or instances.setMetadata. By default, gcloud will add your public key to the project metadata (for all instances), unless the instance to which you’re connecting has been configured to block project wide SSH keys; in that case, it will add the public key to the instance’s metadata. Both API calls are authenticated with the token you received when you did gcloud auth login. The public key is encrypted in transit, because gcloud commands are executed via API calls over HTTPS.

At this point, you have a brand new SSH user private key. Unless you’ve copied it elsewhere, it’s local to your computer. The same is true for its corresponding public key, except that you have shared it with Google by sending it via an authenticated gcloud command over HTTPS. So the only systems that should have access to the public key are the one(s) you’re running in your GCP project.

Only the bearer of the SSH user private key can successfully authenticate to the SSH server(s) running in GCP. Only the SSH server(s) running in GCP have a copy of your SSH user public key. Thus, it’s safe to accept the host public key automatically. If you were connecting to a rogue server instead of your GCP SSH server, the authentication process would fail because that other server would not have your SSH user public key.

There are certain situations in GCP automatically re-creates a SSH server’s host key pair. For example, if you use an instance to create a template from which other instances in a group would be generated. In each case, there’s a need to update the SSH host key. For this reason, performing strict SSH host key checking can generate misleading error messages.

update Sep 5, 2018

It is possible to connect to an instance without "glcoud" and GCP team is aware of and working on it. A public bug is filed here. Any progress can be found there.

Answer 3

No, there is no straightforward way to do this (there is an open issue to remedy this security risk).

Answer 4

Host keys are already retrieved for you to the box you are running gcloud on, in ~/.ssh/google_compute_known_hosts.

Per --help, the behavior of host key checking is as follows:

 --strict-host-key-checking=STRICT_HOST_KEY_CHECKING
    Override the default behavior of StrictHostKeyChecking for the
    connection. By default, StrictHostKeyChecking is set to 'no' the first
    time you connect to an instance, and will be set to 'yes' for all
    subsequent connections. STRICT_HOST_KEY_CHECKING must be one of: yes,
    no, ask.

Which is the behavior in the log file if you look at subsequent connections.

2018-08-04 14:48:29,222 DEBUG    root            Running command [/usr/bin/ssh -t -i /home/john/.ssh/google_compute_engine -o CheckHostIP=no -o HostKeyAlias=compute.1520573357386694112 -o IdentitiesOnly=yes -o StrictHostKeyChecking=no -o UserKnownHostsFile=/home/john/.ssh/google_compute_known_hosts john@35.188.181.248].

2018-08-04 14:51:06,249 DEBUG    root            Executing command: [u'/usr/bin/ssh', u'-t', u'-i', u'/home/john/.ssh/google_compute_engine', u'-o', u'CheckHostIP=no', u'-o', u'HostKeyAlias=compute.1520573357386694112', u'-o', u'IdentitiesOnly=yes', u'-o', u'StrictHostKeyChecking=yes', u'-o', u'UserKnownHostsFile=/home/john/.ssh/google_compute_known_hosts', u'john@35.188.181.248']

This is a convenience so the first connection will not ask about the host, but future ones will check.

Effectively it is the same thing as a user seeing an unknown host key from their client and accepting it without reading the key. With some confidence in the host name of the connection this is reasonable to do in most scenarios.

In this case, your trust in the system should be how much you trust GCE metadata service to route you to the correct host, and push your ssh key to it.