0

I want to add eight public keys via instance metadata to avoid adding them manually (i.e.: ssh to VMs, pasting the keys to .ssh/authorized_keys, etc.).
I added the keys in Terraform (four distinct keys for two users) using the metadata attribute of the google_compute_instance:

resource "google_compute_instance" "host" {
  count         = var.number_of_hosts
  
  // vm details...

  metadata = {
    "ssh-keys" = <<EOF
    user1:${file("${path.root}/key1.pub")}
    user1:${file("${path.root}/key2.pub")}
    user1:${file("${path.root}/key3.pub")}
    user1:${file("${path.root}/key4.pub")}
    user2:${file("${path.root}/key1.pub")}
    user2:${file("${path.root}/key2.pub")}
    user2:${file("${path.root}/key3.pub")}
    user2:${file("${path.root}/key4.pub")}
    EOF
  }

I ran terraform apply. I opened the GCP console and clicked on one of the deployed machines. In the "Details" tab, I can see all eight keys in the SSH Keys tab. Now, when I ssh from my local computer, i.e., ssh user2@EXTERNAL_IP (I deliberately started with user2, not user1 - not a typo) and then cat ~/.ssh/authorized_keys, I can only see the following:

user1 : key1
user2 : key4

Thus, I can't ssh to VM2 because the public part of the key pair that USER 2 has access to is not ~/.ssh/authorized_keys even though it is declared in the instance metadata.

On the other hand, when I do user1@EXTERNAL_IP and cat ~/.ssh/authorized_keys, I can see:

user1 : key1
user2 : key4
user1 : key1 (duplicate)

Since the private key that corresponds to user1 : key1 is there, I can ssh to VM2 successfully.

What baffles me:

  1. Why are not all keys declared in the instance metadata added to the authorized_keys?
  2. Why is there a difference in the content of the authorized_keys depending on the user?
  3. Where does the duplicate come from?

Edit - some additional information:

  1. the image used - ubuntu-minimal-2004-focal-v20230427
  2. ssh_config (only uncommented lines):
Include /etc/ssh/ssh_config.d/*.conf
Host *
SendEnv LANG LC_*
HashKnownHosts yes
GSSAPIAuthentication yes
  1. sshd_config (only uncommented lines):
Include /etc/ssh/sshd_config.d/*.conf
PasswordAuthentication no
ChallengeResponseAuthentication no
UsePAM yes
X11Forwarding yes
PrintMotd no
AcceptEnv LANG LC_*
Subsystem       sftp    /usr/lib/openssh/sftp-server

Edit 2 - relevant fragment of /var/log/syslog

@John Hanley

You were on the right track. I can see some messages of the following form: google_guest_agent[566]: ERROR non_windows_accounts.go:199 Invalid ssh key entry - unrecognized format: in the syslog. However, looking at the form of my keys, I can't entirely agree with a part of your comment: "some public keys have a username appended which will not work". Keys that I read in Terraform and that later appear in the authorized_keys look as follows:

ssh-rsa AAArsT3 username1

or

ecdsa-sha2-nistp256 AAAAE2= username1

Unfortunately, there is still some wild guessing involved. I can now see four keys in the authorized_keys (3x username1, 1x username2). It is better than 2, but I'm asking myself why the four others are still missing. I did a copy-pasting and only changed the user at the end. Is there an agreement on how the keys passed to the GCP via instance-metadata should look (presence/absence of username, newlines, etc.)?

mångata
  • 109
  • 3
  • 1
    My guess is that the public key files are not formatted correctly for Google Cloud metadata. They should have two fields and look like `ssh-rsa B64PUBLICKEY`. Some public keys have a username appended which will not work. – John Hanley May 25 '23 at 21:09
  • If we assume that neither `key3` nor `key4` are properly formatted (as neither of them appears in the `authorized_keys`), it still does not explain why the file only contains `user1 : key1` and `user2 : key4`, instead of `user{1,2} : key1` and `user{1,2} : key4`. Since `key1` and `key4` do appear in the file, I presume that their format is valid. The presence of a duplicate in the second case remains a mystery as well. – mångata May 26 '23 at 06:15
  • 1
    Look at the logs the OS creates (not Cloud Logging). You can see the setup from metadata. Show the relevant log entries in your post. – John Hanley May 26 '23 at 06:34
  • @JohnHanley I took a look at the `syslog` and edited my post. – mångata May 26 '23 at 12:21
  • Remove the trailing username and try again. You are getting an error regarding invalid format ... – John Hanley May 26 '23 at 16:27
  • The GCP docs is kinda misleading then. Here, in the section "Add SSH keys to instance metadata during VM creation", in the "Terraform" tab (https://cloud.google.com/compute/docs/connect/add-ssh-keys#during-vm-creation), we can see the following: ` metadata = { "ssh-keys" = < – mångata May 26 '23 at 17:27
  • Let us [continue this discussion in chat](https://chat.stackexchange.com/rooms/146288/discussion-between-mangata-and-john-hanley). – mångata May 26 '23 at 17:27
  • There is a difference between updating Compute Engine metadata via the GUI or script and updating `ssh-keys` via Terraform. Use the correct format. – John Hanley May 26 '23 at 17:51

0 Answers0