I'm using Telmate's Terraform provider for Proxmox and trying to deploy Flatcar linux virtual machines using Cloud Init by passing files via cicustom. Based on their example, I have crafted the following Terraform file:
variable "pve_user" {
}
variable "pve_password" {
}
variable "pve_host" {
}
provider "proxmox" {
pm_tls_insecure = true
pm_api_url = "https://SNIP/api2/json"
pm_user = "SNIP"
pm_password = "SNIP"
pm_parallel = 4
}
resource "null_resource" "cloud_init_config_files" {
connection {
type = "ssh"
user = var.pve_user
password = var.pve_password
host = var.pve_host
}
provisioner "file" {
source = "./templates/cloud-config.yml"
destination = "/var/lib/vz/snippets/cloud-config.yml"
}
}
resource "proxmox_vm_qemu" "k8s-masters" {
depends_on = [
null_resource.cloud_init_config_files
]
count = 1
name = "VM-${count.index}"
clone = "VM-Template"
full_clone = true
target_node = "192.168.20.10"
pool = "VM"
cores = 2
sockets = 1
memory = 10240
network {
id = 0
model = "virtio"
bridge = "vmbr0"
tag = 50
}
disk {
id = 0
type = "scsi"
size = 30
storage = "Pool"
storage_type = "zfspool"
backup = true
iothread = true
}
onboot = true
agent = 1
os_type = "cloud-init"
ssh_user = "core"
cicustom = "user=local:snippets/cloud-config.yml"
ipconfig0 = "ip=192.168.50.10/24,gw=192.168.50.1"
sshkeys = "ssh-rsa SNIP"
ssh_private_key = <<EOF
-----BEGIN RSA PRIVATE KEY-----
SNIP
-----END RSA PRIVATE KEY-----
EOF
}
After running terraform apply, the VM is successfully created and Flatcar is bootstrapped. If you connect to the console via Proxmox, however, the VM shows that the IP Address is one received via DHCP instead of the one I provided. Furthermore, the SSH key does not work so I am unable to connect to the VM to troubleshoot.
If I comment out the cicustom line, and simply rely on ipconfig0 and the other normal options, the VM comes up and my SSH key does in fact work. However, the specified IP address is still not used; the VM just uses one provided by DHCP instead. Despite being able to access the VM, I still want to use a custom Cloud Init config file so I can have access to more powerful configuration options.
I've tried various combinations of my cloud-config.yml file. Including things as simple as:
hostname: test
to things more detailed like:
storage:
files:
- path: /opt/file1
filesystem: root
contents:
inline: Hello, world!
mode: 0644
user:
id: 500
group:
id: 501
passwd:
users:
- name: core
ssh_authorized_keys:
- ssh-rsa SNIP
Despite all of this, my SSH keys never work after I pass something to cicustom and, based on the fact that the hostname is never manipulated, I'm assuming Cloud Init just outright isn't receiving the custom config file at all. I've tried passing in both Ignition and Container Linux Config formatted files.
Is Flatcar Linux broken with Proxmox/cicustom? Searching Google for things like "flatcar" "proxmox", "coreos" "proxmox", "container linux" "proxmox", "flatcar" "cicustom", etc don't seem to turn up much results. I guess there aren't a lot of people out there bridging Cloud Native operating systems with bare metal? hehehe
What does come up, however, is this interesting script. Lines 104-132 discuss creating a Flatcar/CoreOS template for Proxmox. So someone else has done this before, at least? Who knows if they passed in a custom cloud init config file afterwards?
Any ideas?
EDIT: ------------------------------------------------------------
Added logs from user-configdrive systemd unit when not passing cicustom
Aug 20 16:22:17 localhost coreos-cloudinit[831]: 2020/08/20 16:22:17 Checking availability of "cloud-drive"
Aug 20 16:22:17 localhost coreos-cloudinit[831]: 2020/08/20 16:22:17 Fetching user-data from datasource of type "cloud-drive"
Aug 20 16:22:17 localhost coreos-cloudinit[831]: 2020/08/20 16:22:17 Attempting to read from "/media/configdrive/openstack/latest/user_data"
Aug 20 16:22:17 localhost coreos-cloudinit[831]: 2020/08/20 16:22:17 line 6: warning: unrecognized key "chpasswd"
Aug 20 16:22:17 localhost coreos-cloudinit[831]: 2020/08/20 16:22:17 line 9: warning: incorrect type for "users[0]" (want struct)
Aug 20 16:22:17 localhost coreos-cloudinit[831]: 2020/08/20 16:22:17 line 10: warning: unrecognized key "package_upgrade"
Aug 20 16:22:17 localhost coreos-cloudinit[831]: 2020/08/20 16:22:17 Fetching meta-data from datasource of type "cloud-drive"
Aug 20 16:22:17 localhost coreos-cloudinit[831]: 2020/08/20 16:22:17 Attempting to read from "/media/configdrive/openstack/latest/meta_data.json"
Aug 20 16:22:17 localhost coreos-cloudinit[831]: 2020/08/20 16:22:17 Attempting to read from "/media/configdrive/openstack/content/0000"
Aug 20 16:22:17 localhost coreos-cloudinit[831]: 2020/08/20 16:22:17 Parsing user-data as cloud-config
Aug 20 16:22:17 localhost coreos-cloudinit[831]: 2020/08/20 16:22:17 Merging cloud-config from meta-data and user-data
Aug 20 16:22:17 VM-0 coreos-cloudinit[831]: 2020/08/20 16:22:17 Set hostname to VM-0
Aug 20 16:22:17 VM-0 coreos-cloudinit[831]: 2020/08/20 16:22:17 Authorized SSH keys for core user
Aug 20 16:22:17 VM-0 coreos-cloudinit[831]: 2020/08/20 16:22:17 Failed to apply cloud-config: Invalid option to manage_etc_hosts
Aug 20 16:22:17 VM-0 systemd[1]: user-configdrive.service: Main process exited, code=exited, status=1/FAILURE
EDIT 2: ----------------------------------------------------------
Added the contents of the files referenced by the above Systemd unit.
core@VM-0 ~ $ cat /media/configdrive/openstack/latest/user_data
#cloud-config
hostname: VM-0
manage_etc_hosts: true
ssh_authorized_keys:
- REDACTED
chpasswd:
expire: False
users:
- default
package_upgrade: true
As you pointed out, manage_etc_hosts is set to true here by Proxmox.
core@VM-0 ~ $ cat /media/configdrive/openstack/content/0000
auto lo
iface lo inet loopback
dns_nameservers 192.168.1.100
dns_search example.com
auto eth0
iface eth0 inet static
address 192.168.50.10
netmask 255.255.255.0
gateway 192.168.50.1
This section contains all of the correct network configuration, however it is all overridden by DHCP so it's ignored.
core@VM-0 ~ $ cat /media/configdrive/openstack/latest/meta_data.json
{
"uuid": "e61563c9057e9162c4e14d111fea171379170532",
"network_config": { "content_path": "/content/0000" }
}
EDIT 3: ----------------------------------------------------------
Added my attempt at a custom Cloud Init file
manage_etc_hosts: false
hostname: "test"
ssh_authorized_keys:
- REDACTED
EDIT 4: ----------------------------------------------------------
I followed this guide to basically accomplish the same thing directly with qemu instead of using Proxmox. Using:
This cloud-config.ign:
{
"ignition": { "version": "2.2.0" },
"passwd": {
"users": [
{
"name": "core",
"sshAuthorizedKeys": [
"REDACTED"
]
}
]
},
"storage": {
"files": [{
"filesystem": "root",
"path": "/etc/hostname",
"mode": 420,
"contents": { "source": "data:,test" }
}]
}
}
OR this cloud-config.yml:
hostname: test
passwd:
users:
- name: tj
ssh_authorized_keys:
- REDACTED
both produce the same result. An unconnectable virtual machine that spins up with the hostname localhost.
- Here are the logs that were produced using
cloud-config.ign. - Here are the logs that were produced using
cloud-config.yml.
EDIT 5: ----------------------------------------------------------
These logs are produced by user-configdrive.service when using Telmate's provider against a Flatcar template.
Aug 21 21:41:50 localhost coreos-cloudinit[795]: 2020/08/21 21:41:50 Checking availability of "cloud-drive"
Aug 21 21:41:50 localhost coreos-cloudinit[795]: 2020/08/21 21:41:50 Fetching user-data from datasource of type "cloud-drive"
Aug 21 21:41:50 localhost coreos-cloudinit[795]: 2020/08/21 21:41:50 Attempting to read from "/media/configdrive/openstack/latest/user_data"
Aug 21 21:41:50 localhost coreos-cloudinit[795]: 2020/08/21 21:41:50 Fetching meta-data from datasource of type "cloud-drive"
Aug 21 21:41:50 localhost coreos-cloudinit[795]: 2020/08/21 21:41:50 Attempting to read from "/media/configdrive/openstack/latest/meta_data.json"
Aug 21 21:41:50 localhost coreos-cloudinit[795]: 2020/08/21 21:41:50 Attempting to read from "/media/configdrive/openstack/content/0000"
Aug 21 21:41:50 localhost coreos-cloudinit[795]: Detected an Ignition config. Exiting...
Aug 21 21:41:50 localhost systemd[1]: Started Load cloud-config from /media/configdrive.
EDIT 6: ----------------------------------------------------------
Here is the entire Systemd journal from the first output after using the cloud-config.ign in EDIT 4 with Telmate's provider.
EDIT 7: ----------------------------------------------------------
These qm commands produce a working Flatcar VM using cicustom.
qm create 101 --name test --cores 2 --memory 2048 --net0 "virtio,bridge=vmbr0"
qm set 101 --net0 "virtio,bridge=vmbr0,tag=50"
qm importdisk 101 /mnt/RAIDPool_Templates/template/iso/flatcar_production_qemu_image.img FlashPool
qm set 101 --scsihw virtio-scsi-pci --scsi0 FlashPool:vm-101-disk-0 --ide2 FlashPool:cloudinit --boot c --bootdisk scsi0 --serial0 /dev/tty0 --ipconfig0 ip=dhcp --citype configdrive2
qm set 101 --cicustom user=RAIDPool_Templates:snippets/user-data.yml
qm start 101
ssh -i ~/.ssh/sol.milkyway.kubernetes core@192.168.50.219
user-data.yml:
#cloud-config
hostname: test
manage_etc_hosts: true
ssh_authorized_keys:
- ssh-rsa REDACTED
chpasswd:
expire: False
users:
- default
package_upgrade: true
Passing the same user-data.yml to cicustom within the proxmox_vm_qemu resource doesn't seem to be working though.
