3

I'm trying to set up an on-premises Kubernetes cluster. For starters I used kubeadm to initialize my control plane on one node. But all kube-system pods constantly crash. I took a deep look into the pod logs via crictl and it turns out that most pods crash because they cannot reach the kube-apiserver on <node IP>:6443. And the kube-apiserver seems to crash, because it cannot reach etcd on 127.0.0.1: 2379. And etcd 'crashes' because it receives a sigterm after a couple of minutes without any apparent reason. Using strace on the etcd container process gives:

--- SIGTERM {si_signo=SIGTERM, si_code=SI_USER, si_pid=0, si_uid=0} ---

I've tried flanneld, calico and weave net as CNI, without success. Swap was already disabled from the beginning. Firewall does not have any custom rules. As a kubernetes newbie I cannot think of other possible reasons why this fails and would be grateful for any hints.

System information:

  • Debian 11 on VM, 2 CPUs, 8GB Memory
  • Kubernetes v1.25.0
  • kubeadm, kubectl, kubelet version: 1.25.0-00, all installed via apt following kubeadm installation docs
  • containerd v1.6.8, installed following getting-started guide using binary from releases
    • runc v1.1.4 binary from GitHub
    • containernetworking plugins v1.1.1 from GitHub
  • Cgroup systemd, configured as described here and here
Sai Chandra Gadde
  • 140
  • 7
Be Ku
  • 133
  • 4

2 Answers2

5

Having the same issue with etcd. Are you by any chance using --upload-certs and have a multi-master environment?

Try adding:

GRUB_CMDLINE_LINUX_DEFAULT="systemd.unified_cgroup_hierarchy=0"

To your /etc/default/grub file and then run update-grub, reboot and see if that helps.

Slyke
  • 66
  • 2
  • Thank you so much!! Now everything is running smoothly, and I would never have found this setting. – Be Ku Sep 30 '22 at 07:18
0

I had the same issue, after installing an HA cluster on ubuntu 22.04.2 LTS , although following the same exact steps to configure cgroup on kubernetes and containerd , the newly joined nodes pods were continuosly crashing until i followed the recommendations on slyke

Ayman Abu Qutriyah
  • 1