Running cloud-init twice via Packer & Terraform on VMware Ubuntu 22.04 guest

Question

I would like to create VMs with varying CPU, RAM and network configuration from the same Ubuntu template in ESXi.

When $ packer build -var-file=packer/variables.pkr.hcl -var-file=packer/secret.pkrvars.hcl packer/template.pkr.hcl is run, it reads the following packer/template.pkr.hcl:

variable "vm_name" {
  type = string
  default = "Ubuntu_Server_22.04_LTS"
}

variable "esxi_password" {
  type =  string
  default = "password"
  sensitive = true
}

variable "vm_password" {
  type =  string
  default = "password"
  sensitive = true
}

source "vmware-iso" "ubuntu-2204" {
  vm_name = "${var.vm_name}"
  guest_os_type = "ubuntu-64"

  iso_checksum = "sha256:84aeaf7823c8c61baa0ae862d0a06b03409394800000b3235854a6b38eb4856f"
  iso_url = "https://REDACTED/ubuntu-22.04-live-server-amd64.iso"

  http_directory = "/home/REDACTED/packer/http"

  shutdown_command = "sudo shutdown -P now"

  remote_type = "esx5"

  remote_datastore = "REDACTED"
  remote_host = "REDACTED"
  remote_username = "REDACTED"
  remote_password = "${var.esxi_password}"
  remote_private_key_file = ""

  cpus = 8
  memory = 16384
  disk_size = 16384

  network_adapter_type = "vmxnet3"
  network_name = "REDACTED"

  headless = false
  vnc_over_websocket = true
  insecure_connection = true

  tools_upload_flavor = "linux"

  skip_export = true
  keep_registered = true

  ssh_username = "REDACTED"
  ssh_password = "${var.vm_password}"
  ssh_timeout = "15m"
  ssh_handshake_attempts = "100"

  boot_wait = "3s"
  boot_command = [
    "<esc><esc><esc><esc>e<wait>",
    "<del><del><del><del><del><del><del><del><del><del>",
    "<del><del><del><del><del><del><del><del><del><del>",
    "<del><del><del><del><del><del><del><del><del><del>",
    "<del><del><del><del><del><del><del><del><del><del>",
    "<del><del><del><del><del><del><del><del><del><del>",
    "<del><del><del><del><del><del><del><del><del><del>",
    "<del><del><del><del><del><del><del><del><del><del>",
    "<del><del><del><del><del><del><del><del><del><del>",
    "<del><del><del><del><del><del><del><del><del><del>",
    "<del><del><del><del><del><del><del><del><del><del>",
    "<del><del><del><del><del><del><del><del><del><del>",
    "<del><del><del>",
    "linux /casper/vmlinuz --- autoinstall ds=\"nocloud-net;seedfrom=http://[{{.HTTPIP}}]:{{.HTTPPort}}/\"<enter><wait>",
    "initrd /casper/initrd<enter><wait>",
    "boot<enter>",
    "<enter><f10><wait>"
  ]
}

build {
  sources = ["sources.vmware-iso.ubuntu-2204"]
  provisioner "shell" {
    inline = [
      "ls /"
    ]
  }
}

packer/http/user-data contains the following:

#cloud-config
autoinstall:
  version: 1
  early-commands:
    # Stop SSH to prevent Packer from connecting too early
    - systemctl stop ssh
  apt:
    preserve_sources_list: false
    primary:
    - arches: [amd64, i386]
      uri: https://REDACTED
    - arches: [default]
      uri: http://ports.ubuntu.com/ubuntu-ports
  locale: en_US
  keyboard:
    layout: en
    variant: us
  network:
    version: 2
    renderer: networkd
    ethernets:
      ens160:
        dhcp4: true
        dhcp-identifier: mac
        dhcp6: true
  storage:
    layout:
      name: direct
    config:
      - type: disk
        id: disk0
        match:
          size: largest
      - type: partition
        id: boot-partition
        device: disk0
        size: 500M
      - type: partition
        id: root-partition
        device: disk0
        size: -1
  ssh:
    install-server: true
    allow-pw: true
    authorized-keys:
      - ssh-ed25519 REDACTED
  identity:
    hostname: ubuntu
    username: REDACTED
    password: REDACTED
  packages:
    - open-vm-tools
    - python3
  late-commands:
    - echo 'REDACTED ALL=(ALL) NOPASSWD:ALL' > /target/etc/sudoers.d/REDACTED
    - curtin in-target --target=/target -- chmod 440 /etc/sudoers.d/REDACTED
    - curtin in-target --target=/target -- apt-get update
    - curtin in-target --target=/target -- apt-get upgrade --yes
    - curtin in-target --target=/target -- sudo cloud-init clean

This creates an Ubuntu 22.04 Server template which I can then use Terraform to provision virtual machines from.

# cat /var/log/installer/autoinstall-user-data run on the VM shows that Packer has successfully provided user-data. It has been executed, proven by my ability to login via SSH.

When $ terraform apply -var-file=secret.tfvars is run within my terraform directory, it reads the following main.tf with provider "esxi" provided by https://github.com/josenk/terraform-provider-esxi:

variable "vm_name" {
  description = "The name of the virtual machine"
  default     = "ubuntu-terraformed"
  type        = string
}

variable "esxi_password" {
  description = "The password for the ESXi root user"
  type        = string
}

provider "esxi" {
  esxi_hostname = "REDACTED"
  esxi_username = "REDACTED"
  esxi_password = "${var.esxi_password}"
}

data "template_file" "Test" {
  template = file("userdata.tpl")
  vars = {
    HOSTNAME = "${var.vm_name}"
  }
}

resource "esxi_guest" "Test" {
  guest_name = "${var.vm_name}"
  disk_store = "REDACTED"

  clone_from_vm = "Ubuntu_Server_22.04_LTS"

  network_interfaces {
    virtual_network = "REDACTED"
    nic_type = "vmxnet3"
  }

  network_interfaces {
    virtual_network = "REDACTED"
    nic_type = "vmxnet3"
  }

  network_interfaces {
    virtual_network = "REDACTED"
    nic_type = "vmxnet3"
  }

  network_interfaces {
    virtual_network = "REDACTED"
    nic_type = "vmxnet3"
  }

  network_interfaces {
    virtual_network = "REDACTED"
    nic_type = "vmxnet3"
  }

  network_interfaces {
    virtual_network = "REDACTED"
    nic_type = "vmxnet3"
  }

  network_interfaces {
    virtual_network = "REDACTED"
    nic_type = "vmxnet3"
  }

  guestinfo = {
    "userdata.encoding" = "gzip+base64"
    "userdata"          = base64gzip(data.template_file.Test.rendered)
  }
}

userdata.tpl contains the following:

#cloud-config

hostname: ${HOSTNAME}
network:
  version: 2
  renderer: networkd
  ethernets:
    ens160:
      dhcp4: true
      dhcp-identifier: mac
      dhcp6: true
    ens161:
      dhcp4: true
      dhcp-identifier: mac
      dhcp6: true
    ens192:
      dhcp4: true
      dhcp-identifier: mac
      dhcp6: true
    ens193:
      dhcp4: true
      dhcp-identifier: mac
      dhcp6: true
    ens224:
      dhcp4: true
      dhcp-identifier: mac
      dhcp6: true
    ens225:
      dhcp4: true
      dhcp-identifier: mac
      dhcp6: true
    ens256:
      dhcp4: true
      dhcp-identifier: mac
      dhcp6: true
package_upgrade: true
#ntp:
#  enabled: true
#  servers:
#    - REDACTED
#timezone: REDACTED
#late-commands:
#  - curtin in-target --target=/target -- sudo sed -i 's/#NTP=/NTP=REDACTED/g' /etc/systemd/timesyncd.conf
#  - curtin in-target --target=/target -- sudo timedatectl set-ntp true
#  - curtin in-target --target=/target -- sudo timedatectl set-timezone REDACTED
#  - curtin in-target --target=/target -- sudo systemctl restart systemd-timesyncd.service

This creates a VM based on the packer template with the correct guest parameters. The VMware guest configuration contains the userdata property which I have verified matches the supplied user-data after base64 decoding and ungzipping the guest parameter.

The issue I experience is that the VM does not seem to contain or execute the "second" Terraform cloud-init userdata.

/var/lib/cloud/instance/user-data.txt shows none of the second configuration:

#cloud-config
growpart:
  mode: 'off'
locale: en_US.UTF-8
preserve_hostname: true
resize_rootfs: false
ssh_pwauth: true
users:
- gecos: me
  groups: !!set
    adm: null
    cdrom: null
    dip: null
    lxd: null
    plugdev: null
    sudo: null
  lock_passwd: false
  name: me
  passwd: REDACTED
  shell: /bin/bash
  ssh_authorized_keys:
  - ssh-ed25519 REDACTED
    me

/var/log/cloud-init-output.log shows:

schema.py[WARNING]: Invalid cloud-config provided:
users.0: {'gecos': 'me', 'groups': {'sudo', 'lxd', 'cdrom', 'adm', 'plugdev', 'dip'}, 'lock_passwd': False, 'name': 'me', 'passwd': 'REDACTED', 'shell': '/bin/bash', 'ssh_authorized_keys': ['ssh-ed25519 REDACTED me']} is not valid under any of the given schemas

Is attempting to apply user-data twice, when my workflow is Packer generating a template, and Terraform applying a new VM with that template, correct? If so, where am I best to find out why Ubuntu does not contain and execute the second iteration?

As an aside, if there are any other recommendations I should consider please feel free to comment. I imagine I should be able to achieve this with the upcoming plan to implement Ansible, but I still need to set options such as Hostname.

The `cloud-init` for both Packer and Terraform should execute correctly. You would probably want to check the logs on the system after infra provisioning to troubleshoot the second `cloud-init`. — Matthew Schuchard, Jun 10 '22 at 10:48
Thanks @MattSchuchard, I thought so initially as well. I've added some additional logs that I believe may be related while I continue to investigate. — abestic9, Jun 25 '22 at 00:16

score 4 · Accepted Answer · answered Jun 27 '22 at 07:41

I was able to resolve this by changing the Packer build to the following:

build {
  sources = ["sources.vmware-iso.ubuntu-2204"]
  provisioner "shell" {
    inline = [
      "sudo rm -f /etc/cloud/cloud.cfg.d/99-installer.cfg",
      "sudo rm -f /etc/cloud/cloud.cfg.d/subiquity-disable-cloudinit-networking.cfg",
      "echo 'disable_vmware_customization: false' | sudo tee -a /etc/cloud/cloud.cfg",
      "sudo sed -i 's|nocloud-net;seedfrom=http://.*/|vmware|' /etc/default/grub",
      "sudo update-grub",
      "sudo cloud-init clean"
    ]
  }
}

The cloud.cfg adjustments are referenced in the following articles, and allow OS network configuration via cloud-init which was the final step in getting this all to work:

Adjusting the GRUB boot command, as well as appending cloud-init clean, to the Packer build provisioning step ensures that just before final shutdown of the Packer-built guest, the VM will no longer attempt to boot from the initial preseed, and instead will use the VM guestinfo specified on next boot.

I also split the Terraform user-data out into meta-data for the instance identification and network configuration, leaving the now-uncommented NTP and timezone configuration in user-data.

metadata.tpl:

#cloud-config

instance-id: ${HOSTNAME}
local-hostname: ${HOSTNAME}
network:
  version: 2
  ethernets:
    nics:
      match:
        name: ens*
      dhcp-identifier: mac
      dhcp4: yes
      dhcp6: yes

userdata.tpl:

#cloud-config

package_upgrade: true
ntp:
  enabled: true
  servers:
    - REDACTED
timezone: REDACTED

As an aside, a simpler way to configure network interfaces was to use the match directive instead of explicitly specifying interfaces.

All items are now configured as intended. It is possible this configuration is only required in ESXi using vmware-iso, as I have not seen these steps required elsewhere.

The gecos warning appears to have been a red herring and I've made no further attempts to resolve that.

Running cloud-init twice via Packer & Terraform on VMware Ubuntu 22.04 guest

1 Answers1