0

I've been to countless sites including this one, trying to figure out how to make LXD run a cloud-config when I launch my LXD container. Some places recommend setting up a profile (didn't work). Others recommend redirecting a YAML file into the lxc command (didn't work), and others recommend using the --config option and passing the file in that way (didn't work). Some places say I have to add #cloud-config to my config, others don't bother. Some even recommend using an XML file. I'm obviously missing some critical piece of information that everyone else just does by default, but I can't figure out what it is.

My updated "simple example" that tries to install tree and tries to touch /run/cloud-config-did-run:

lxc delete -f x

cat << EOF >config.yml
#cloud-config
output: {all: '| tee -a /var/log/my-cloud-init-output.log'}
package_update: true
package_upgrade: true
package_reboot_if_required: true
packages:
  - tree
runcmd:
  - touch /run/cloud-config-did-run
EOF

lxc launch ubuntu: x --config=user.user-data="$(cat config.yml)"
sleep 5
lxc exec x -- bash -c "ls /run"
lxc exec x -- bash -c "tree /etc"

The output DOES get directed to /var/log/my-cloud-init-output.log, so it is getting processed, but nothing other than the output directive runs (the logs don't even mention any other things running, or any errors - just the standard SSH keygen stuff).

Maybe the indentation is wrong? Or the config is in the wrong subtree? Or there's some magic value missing? A version of LXD where this is broken? (I'm running version 4.20). I've been at this for 10 hours so far and no matter what I do, my cloud config is completely ignored (no errors, no logs, no record of anything running, no record that I ever instructed it to do anything - other than the standard ssh keygen stuff that's apparently baked in). Can someone please turn the above into a working example that is guaranteed to run if I just paste it into a shell?

Karl
  • 239
  • 1
  • 3
  • 8
  • im not sure, but what are you trying to do or fix? – djdomi Dec 19 '21 at 18:08
  • I'm trying to make it auto-run a command inside of an LXD container when I launch it. Evenrually I want it to auto-install some packages, but for now I just want something, anything to work. – Karl Dec 19 '21 at 18:12
  • Have you checked `/var/log/cloud-init.log` ? – AlexD Dec 19 '21 at 18:59
  • Yes. That's where the above log entry comes from. There are no error entries or anything that would suggest anything going wrong. – Karl Dec 19 '21 at 19:00
  • There is a note in `cloud-init` `runcmd` module docs: "when writing files, do not use /tmp dir as it races with systemd-tmpfiles-clean LP: #1707222. Use /run/somedir instead." – AlexD Dec 19 '21 at 19:02
  • Installing packages and outputting status text also doesn't work. I was just using this as an example, because literally nothing works. – Karl Dec 19 '21 at 19:12
  • You can try adding `output: {all: '| tee -a /var/log/cloud-init-output.log'}` to `config.yml`. It should help a bit with debugging. – AlexD Dec 19 '21 at 19:18
  • Updated to use /run and also attempt to install something. Strangely, the `output` directive is honored, but nothing else is, and nothing out of the ordinary shows up in the logs. – Karl Dec 19 '21 at 19:40
  • You probably want to check if `cloud-init` recognizes that it is being run at the `first-boot` stage. If I remember correctly it skips most modules after the first boot. – AlexD Dec 19 '21 at 20:01
  • How would I do that? I'm coming at this totally new, so I've been following blogs and howtos, but none of them have worked, and none of them tell how to troubleshoot because this is supposed to be such a trivial thing... I really don't get how it could fail so spectacularly on two separate systems even (one running ubuntu, the other running nixos) – Karl Dec 19 '21 at 20:04
  • This is why I'm hoping that someone would run something trivial that works on their system and then post it here so that I have something verified running elsewhere. – Karl Dec 19 '21 at 20:10

1 Answers1

0

The available (non-)information about cloud-config is really a pain. And the docs are still bad. But first things first:
I tried exactly your code and succeeded (package upgrades + /run/cloud-config-did-run worked). Tested with Ubuntu 20.04 as well as 22.04.

BUT, I work on LXD 5.1. So this might be the important key difference.

Some remarks which might be useful:

  • I configure a simple output to differentiate between the stages init, config, final and therein between standard log and errors. That makes debugging easier.

    output:
  init: ["> /tmp/cloud-init.log", "> /tmp/cloud-init.err"]
  config: [ "> /tmp/cloud-init-config.log", "> /tmp/cloud-init-config.err" ]
  final: [ "> /tmp/cloud-init-final.log", "> /tmp/cloud-init-final.err" ]

  • As soon as I have fired the lxc launch command
    I change into the container with lxc shell x.
    With cloud-init status --wait I look if everything works ("done") or fails.
    If it fails /tmp/*.err is your friend

  • Until now I never had an issue with writing to /tmp. Cloud-init changed the order of execution somewhere in the past. That might have solved it.

  • useless knowledge: package_update is included in package_upgrade

  • #cloud-config is needed. I tried without and it did not work. The YAML was ignored altogether

  • About the cloud-config format: it is YAML (the file extension does not matter). I don't think anything else works for LXD - besides a MIME multi-part archive

  • MIME multi-part archives work with LXD and its a good way to reuse code for different configs. Its easy to use but the docs seem rudimentary.
    A little example with your config.yml together with a shell script:

    cat << EOF >test.sh
#!/usr/bin/env bash
touch /run/here-we-go-with-a-shellscript
EOF

cloud-init devel make-mime -a config.yml:cloud-config -a test.sh:x-shellscript-per-once > newconfig
lxc launch ubuntu:j f --config=user.user-data="$(cat /root/newconfig)"
lxc shell f
cloud-init status --wait
ls /run

    A good starting point for working with MIME multi-part archives is probably this cloudinit doc page

  • LXD Profiles with cloud-config worked when we tested it, but we don't use it

I hope that helps somehow. If you are interested I could add a larger cloud-init yaml which we tested for setting up a typical web container behind a reverse proxy.

SeparateReality
  • 101
  • 2