Cilium installation on rancher-desktop

Question

I am not really sure how to formulate properly this question, really beginner with k8s. I am creating a playground for k3s on a laptop, wanting to install cilium and prometheus/grafana monitoring. For that purpose i installed rancher-desktop which created sandboxed environment that runs k3s inside a vm (running on a laptop)

I installed cilium on rancher-desktop v1.0.0 with the helm:

helm install cilium cilium/cilium --version 1.11.1 \
   --namespace kube-system \
   --set prometheus.enabled=true \
   --set operator.prometheus.enabled=true \
   --set hubble.enabled=true \
   --set hubble.metrics.enabled="{dns,drop,tcp,flow,port-distribution,icmp,http}"

It installs, but cilium container fails to start:

Error: failed to generate container "0fae98546697febc25abb4ac49d5e5a2f27a3ee1781bade900f2c767f8d6df28" spec: failed to generate spec: path "/run/cilium/cgroupv2" is mounted on "/run/cilium/cgroupv2" but it is not a shared or slave mount

That leads me to believe that bpf is not mounted. Now, I am versed in Linux but really a novice with k8s. The rancher-desktop packs k3s and starts it in a vm (my pc is Ubuntu 20.04 based). So, qemu starts this vm (lima-rancher-desktop) and I can login into it. I assumed that BPF should be enabled in that VM (it is not). But, maybe I am wrong. Maybe it is inside some containers inside k3s? I enabled BPF even on my laptop, but this doesn't help, k3s runs inside that VM, Here are the pods:

kubectl get pods -A
NAMESPACE           NAME                                      READY   STATUS                      RESTARTS   AGE
kube-system         local-path-provisioner-84bb864455-z2659   1/1     Running                     0          4h28m
kube-system         helm-install-traefik-crd--1-dxcg7         0/1     Completed                   0          4h28m
kube-system         svclb-traefik-7kqgd                       2/2     Running                     0          4h28m
kube-system         helm-install-traefik--1-lbjhw             0/1     Completed                   1          4h28m
kube-system         metrics-server-ff9dbcb6c-rmvd9            1/1     Running                     0          4h28m
kube-system         traefik-55fdc6d984-zpk5s                  1/1     Running                     0          4h28m
cilium-monitoring   prometheus-655fb888d7-mbnb9               1/1     Running                     0          3h52m
cilium-monitoring   grafana-5747bcc8f9-rj5jk                  1/1     Running                     0          3h52m
kube-system         cilium-operator-5ffd7d9795-ktldm          0/1     Pending                     0          3m26s
kube-system         cilium-operator-5ffd7d9795-b8ls9          1/1     Running                     0          3m26s
kube-system         cilium-d5xr4                              0/1     Init:CreateContainerError   0          3m26s
kube-system         coredns-96cc4f57d-r99zl                   1/1     Running                     0          7s

I would appreciate if someone can explain where should BPF be mounted: inside that VM, or inside some container on k3s, and how to mount it?

Note: it doesn't mount in VM anyway:

mount --bind /var/run/bpf /var/run/bpf

has no effect whatsoever and doesn't mount anything, as if it is read-only.

Answer 1

Full solution based on the findings above: https://github.com/cilium/cilium/issues/18675#issuecomment-1050234756

Summary:

Ok so mounting groups myself and making it shared was easy enough.

So in total:

sudo mount bpffs -t bpf /sys/fs/bpf
sudo mount --make-shared /sys/fs/bpf
sudo mkdir -p /run/cilium/cgroupv2
sudo mount -t cgroup2 none /run/cilium/cgroupv2
sudo mount --make-shared /run/cilium/cgroupv2/

From inside the guest, I made a script, which I just invoke via limactl (the $HOME on the host is mounted into the guest):

(⎈ |rancher-desktop:default) ~/g/s/g/c/cilium ❯❯❯ cat setup-cilium-rancher.sh                                                                                                                 fix_grep ◼
#!/bin/sh

set -e

echo Mounting bpf
mount bpffs -t bpf /sys/fs/bpf
mount --make-shared /sys/fs/bpf

echo Mounting cgroups v2 to /run/cilium/cgroupv2
mkdir -p /run/cilium/cgroupv2
mount -t cgroup2 none /run/cilium/cgroupv2
mount --make-shared /run/cilium/cgroupv2/
(⎈ |rancher-desktop:default) ~/g/s/g/c/cilium ❯❯❯ LIMA_HOME="$HOME/Library/Application Support/rancher-desktop/lima" "/Applications/Rancher Desktop.app/Contents/Resources/resources/darwin/lima/bin/limactl" shell 0 sudo sh $PWD/setup-cilium-rancher.sh
Mounting bpf
Mounting cgroups v2 to /run/cilium/cgroupv2

Then I just did a basic helm install, and we have the cilium DaemonSet pod running without problems:

(⎈ |rancher-desktop:default) ~/g/s/g/c/cilium ❯❯❯ helm install cilium cilium/cilium --version 1.11.2 --namespace kube-system                                                                  fix_grep ◼
W0224 12:24:45.168567   78675 warnings.go:70] spec.template.spec.affinity.nodeAffinity.requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms[1].matchExpressions[0].key: beta.kubernetes.io/os is deprecated since v1.14; use "kubernetes.io/os" instead
W0224 12:24:45.168579   78675 warnings.go:70] spec.template.metadata.annotations[scheduler.alpha.kubernetes.io/critical-pod]: non-functional in v1.16+; use the "priorityClassName" field instead
NAME: cilium
LAST DEPLOYED: Thu Feb 24 12:24:44 2022
NAMESPACE: kube-system
STATUS: deployed
REVISION: 1
TEST SUITE: None
NOTES:
You have successfully installed Cilium with Hubble.

Your release version is 1.11.2.

For any further help, visit https://docs.cilium.io/en/v1.11/gettinghelp

(⎈ |rancher-desktop:default) ~/g/s/g/c/cilium ❯❯❯ kubectl get pods -n kube-system                                                                                                                   fix_grep ◼
NAME                                      READY   STATUS      RESTARTS   AGE
local-path-provisioner-84bb864455-57c46   1/1     Running     0          6m20s
helm-install-traefik-crd--1-jdxsr         0/1     Completed   0          6m21s
metrics-server-ff9dbcb6c-ft2c4            1/1     Running     0          6m20s
helm-install-traefik--1-nflmd             0/1     Completed   2          6m21s
svclb-traefik-tgr4h                       2/2     Running     0          6m
traefik-55fdc6d984-l2skq                  1/1     Running     0          6m
cilium-operator-6d8799bcbb-f74g8          0/1     Pending     0          3m15s
cilium-operator-6d8799bcbb-njzk2          1/1     Running     0          3m15s
cilium-h7qzw                              1/1     Running     0          3m15s
coredns-96cc4f57d-zkjlg                   1/1     Running     0          12s

Answer 2

After playing around, I found out it has to be mounted NOT on the host but on the VM. If the BPF FS is already mounted, but not shared, it has to be unmounted then mounted again as shared:

sudo mount bpffs -t bpf /sys/fs/bpf
sudo mount --make-shared /sys/fs/bpf