Failed to validate kubelet flags: the container runtime endpoint address was not specified or empty, use --container-runtime-endpoint to set

Question

I am trying to set up a Kubernetes master node. Every time I try to start kubelet I am getting the error message:

command failed" err="failed to validate kubelet flags: the container runtime endpoint address was not specified or empty, use --container-runtime-endpoint to set

I tries to set up the container runtime endpoint with the following command:

sudo kubelet --container-runtime-endpoint=unix:///run/containerd/containerd.sock

But when I do, I get the following log with a failing command:

I0116 09:43:02.562785    4142 server.go:412] "Kubelet version" kubeletVersion="v1.26.0"
I0116 09:43:02.563067    4142 server.go:414] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
I0116 09:43:02.563458    4142 server.go:575] "Standalone mode, no API client"
I0116 09:43:02.566523    4142 server.go:463] "No api server defined - no events will be sent to API server"
I0116 09:43:02.566664    4142 server.go:659] "--cgroups-per-qos enabled, but --cgroup-root was not specified.  defaulting to /"
I0116 09:43:02.567002    4142 container_manager_linux.go:267] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
I0116 09:43:02.567130    4142 container_manager_linux.go:272] "Creating Container Manager object based on Node Config" nodeConfig={RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: KubeletOOMScoreAdj:-999 ContainerRuntime: CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:cgroupfs KubeletRootDir:/var/lib/kubelet ProtectKernelDefaults:false NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: ReservedSystemCPUs: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[] SystemReserved:map[] HardEvictionThresholds:[]} QOSReserved:map[] CPUManagerPolicy:none CPUManagerPolicyOptions:map[] ExperimentalTopologyManagerScope:container CPUManagerReconcilePeriod:10s ExperimentalMemoryManagerPolicy:None ExperimentalMemoryManagerReservedMemory:[] ExperimentalPodPidsLimit:-1 EnforceCPULimits:true CPUCFSQuotaPeriod:100ms ExperimentalTopologyManagerPolicy:none ExperimentalTopologyManagerPolicyOptions:map[]}
I0116 09:43:02.567232    4142 topology_manager.go:134] "Creating topology manager with policy per scope" topologyPolicyName="none" topologyScopeName="container"
I0116 09:43:02.567305    4142 container_manager_linux.go:308] "Creating device plugin manager"
I0116 09:43:02.567449    4142 state_mem.go:36] "Initialized new in-memory state store"
E0116 09:43:02.570133    4142 run.go:74] "command failed" err="failed to run Kubelet: validate service connection: CRI v1 runtime API is not implemented for endpoint \"unix:///run/containerd/containerd.sock\": rpc error: code = Unimplemented desc = unknown service runtime.v1.RuntimeService"

Can anyone help me with this?

I have already tried resetting kubeadm because I read in a forum that could cause this problem.

This did not solve my problem. Outside of that, I read that going back to an older version of Kubernetes will work. (But I did not try that. I want to use version 1.26 which is the newest).

And I cannot imagine Kubernetes making a version where containerd doesn't work. So where is the fix?

Answer 1

Looks like you are encountering the problem with Removal of the CRI v1alpha2 API and containerd 1.5 support in K8s 1.26.

Possible workarounds:

You can solve this by removing the containerd package that came with Ubuntu and installing the containerd.io package from docker repositories. Then you get a 1.6 release. Then re-generate the container's config including the group changes and restart on 1.25-05. Then you are able to complete the upgrade to 1.26.

From there it was a simple do-release-upgrade to the latest ubuntu.

1) In the Docker repos, there are packages for containerd 1.6 and above. So you can also add the Docker repos, and install containerd.io from there:

sudo mkdir -p /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/docker.gpg
echo   "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update
sudo apt install containerd.io

2) Manually upgrading containerd to 1.6 or above, by downloading and replacing the binaries

wget https://github.com/containerd/containerd/releases/download/v1.6.12/containerd-1.6.12-linux-amd64.tar.gz
tar xvf containerd-1.6.12-linux-amd64.tar.gz
systemctl stop containerd
cd bin
cp * /usr/bin/
systemctl start containerd

3) The one listed in the link above - running an older version of the kubelet (1.25)

apt remove --purge kubelet
apt install -y kubeadm kubelet=1.25.5-00

Please go through the similar ServerFault Answers for detailed step by srep information.

EDIT :

4) Third-party replacement, cri-dockerd, is available. The cri-dockerd adapter lets you use Docker Engine through the Container Runtime Interface.

If you already use cri-dockerd, you aren't affected by the dockershim removal. Before you begin, Check whether your nodes use the dockershim.