kubelet reading from wrong config file?

Question

When I run kubelet version I get an error message ending in:

error: failed to run Kubelet: failed to create kubelet: misconfiguration: kubelet cgroup driver: "cgroupfs" is different from docker cgroup driver: "systemd"

But when I check the config file located at /etc/systemd/system/kubelet.service.d/10-kubeadm.conf, I see the value IS systemd. I have changed the value and done a systemctl daemon-reload and systemctl restart kubelet in between each change and the error message is always the same.

I am guessing it must be reading from the wrong config file, but how can I find where it is reading from!

and the output from `systemctl cat kubelet.service` (as it may be pulling in yet another drop-in)? Plus, don't overlook the value of `kubelet --v=100` to show _exactly_ what's going on — mdaniel, Nov 08 '17 at 06:17

Yu Jiaao · Answer 1 · 2019-10-22T11:03:01.407

try this:

kubelet  --cgroup-driver=systemd version

The "docker" package (1.13.1) already has "systemd" as the default cgroup-driver, see this.

The file driver is systemd changed by default cgroupfs, and docker file driver we installed is systemd caused by inconsistency, which causes the image to fail to start.

     docker info 

...
     Cgroup Driver: systemd

There are two ways now, one is to modify docker, the other is to modify kubelet,

Modify docker: # Modify or create /etc/docker/daemon.json and add the following:

{
  "exec-opts": ["native.cgroupdriver=systemd"]
}

Restart docker:

systemctl restart docker
systemctl status docker

Modify kubelet: #

vim /etc/systemd/system/kubelet.service.d/10-kubeadm.conf

# Note: This dropin only works with kubeadm and kubelet v1.11+
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
# This is a file that "kubeadm init" and "kubeadm join" generates at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably, the user should use
# the .NodeRegistration.KubeletExtraArgs object in the configuration files instead. KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/sysconfig/kubelet
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS

Add the following content --cgroup-driver=systemd to $KUBELET_EXTRA_ARGS.

Or:

$ DOCKER_CGROUPS=$(docker info | grep 'Cgroup' | cut -d' ' -f3)
$ echo $DOCKER_CGROUPS

$ cat >/etc/sysconfig/kubelet<<EOF
KUBELET_CGROUP_ARGS="--cgroup-driver=$DOCKER_CGROUPS"
EOF

#restart
$ systemctl daemon-reload
$ systemctl enable kubelet && systemctl restart kubelet

Or:

DOCKER_CGROUPS=$(docker info | grep 'Cgroup' | cut -d' ' -f3)
echo $DOCKER_CGROUPS
cat >/etc/sysconfig/kubelet<<EOF
KUBELET_EXTRA_ARGS="--cgroup-driver=$DOCKER_CGROUPS"
EOF

# restart
$ systemctl daemon-reload
$ systemctl enable kubelet && systemctl restart kubelet

kubelet reading from wrong config file?

1 Answers1