0

Had my ssh host key reset by GCE. Found

/var/lib/cloud/instances/iid-datasource-none

was created.

https://cloudinit.readthedocs.io/en/latest/topics/datasources/fallback.html?highlight=iid-datasource-none

is not enlightening as to cause / prevention. Anyone know how this aspect of cloudinit works?

user1133275
  • 219
  • 1
  • 11
  • That shouldn't be happening on GCE unless you specifically configured your datasource to do so (you would know if you did this). I suggest running `cloud-init collect-logs` with a `-u` if you have no sensitive userdata, then creating a bug at https://bugs.launchpad.net/cloud-init and attaching the tarball to the issue. – falcojr Feb 23 '22 at 18:51
  • What do you mean reset? SSH host key was recreated? Did you perform any actions on that VM lately? How did you create this VM (image version). – PjoterS Feb 24 '22 at 10:23
  • @PjoterS yes recreated. Stock Google img. I briefly made iptables-persist unable to run, but that did not break a server with the same setup on AWS. I think Google's cloudinit in snapd is not robust but I'd like to find a fix. – user1133275 Feb 25 '22 at 04:38
  • Is it possible to provide some logs? Also it was one time occurrence or this is something which occured a few times? What OS it was? Ubuntu, Debian, RH? – PjoterS Feb 28 '22 at 09:20

1 Answers1

1

iid-datasource-none is the ("fake") instance id reported by the datasource None.

Host key is recreated(like everything else) when cloud init finds the instance_id changed or it can't find data from a previous run. If you recreated the instance this is the expected behavior: host keys are suposed to be specific for a instance. See below how to get different behavior.

How does this work and what is this:

Cloud init tries to do the basic configuration of the instance. To do so it needs some information about what needs to be configured and to which settings? That information comes from a datasource. Different datasources are possible - depending on your needs. Among them DataSourceGCE and DataSourceNone. In the configuration, cloud init has a datasource_list, which is an array of datsources to try. Then there is ds-identify.cfg that tells cloud init how to test and choose the datasource.

In your case the test resulted in chosing DatasourceNone which is kind of a fallback when there is no other Datasource. As this is a (kind of a) fallback, it always returns the same hard coded instance id.

Why would cloud init chose it

It did not find any other enabled and working datasource in the datasource_list.

Likely cause:

  • The image that was used to stand up the instance does not have DataSourceGCE in its datasource_list.
  • The DataSourceGCE was disabled.
  • The Kernel has a ds= option that overrules other settings.

Likely Wokaround

Have a look at /etc/cloud/cloud.cfg and additional files in /etc/cloud/cloud.cfg.d/ Check if any of the files has a "datasource_list:" option in it. Ensure it contains GCE like below:

datasource_list: ['GCE','None']

If no file with such a line exist create /etc/cloud/cloud.cfg.d/datasource.cfg and with the above content. On a shell on the instance do a:

cloud-int clean
reboot

You should end up with the instance id that comes from google. If that does not work check if you have otherwise disabled it like in ds_identify and that gce meta-data is accessible from you instance.

If it works recreate a image with the new settings and use that.

For the host keys

create a file /etc/cloud/cloud.cfg.d/ssh.cfg put in

ssh_keys:
   rsa_public: |
       <your rsa hostkey>
   rsa_private: |
       <your rsa hostkey>
   ecdsa_public: |
       <your ecdsa hostkey>
   ecdsa_private: |
       <your ecdsa hostkey>
   ed25519_public: |
       <your ed25519 hostkey>
   ed25519_private: |
       <your ed25519 hostkey>

Its Yaml so make sure that indentation is correct. You can generate the keys with ssh-keygen or copy the content from the respective files in /etc/ssh of the instance(so any recreated instance gets the same keys)

cloud-init clean 
reboot

... and create an image to persist your changes.

Instead of /etc/cloud/cloud.cfg.d/ you can put the lines to configure hostkeys as well in user_data(or a part if multipart) that starts with "#cloud-config".

Too complicated - simple solution:

Find an image with a correctly preconfigured cloud init and stand up your instance with that image. Use an image without bugs.

kai
  • 111
  • 3