How to setup cri-o with kubeadm and kubelet on Kubernetes 1.18.2?

Question

I am relatively new to Kubernetes and although that I am able to launch the master node (join workers / master nodes) by using the default socket (/var/run/dockershim.sock) I would like to use the cri-o socket (unix:///var/run/crio/crio.sock).

I have been reading any documentation that I was able to find but none it seems to be working for me.

I am running Kubernetes on Centos7.

CRI-O:

# crio version
Version:       1.18.2
GitCommit:     754d46b53595cf2db74d2a73a685d573910b814e
GitTreeState:  clean
BuildDate:     2020-06-25T09:23:58Z
GoVersion:     go1.13.6
Compiler:      gc
Platform:      linux/amd64
Linkmode:      dynamic

Docker:

# docker version
Client: Docker Engine - Community
 Version:           19.03.12
 API version:       1.40
 Go version:        go1.13.10
 Git commit:        48a66213fe
 Built:             Mon Jun 22 15:46:54 2020
 OS/Arch:           linux/amd64
 Experimental:      false

Server: Docker Engine - Community
 Engine:
  Version:          19.03.12
  API version:      1.40 (minimum version 1.12)
  Go version:       go1.13.10
  Git commit:       48a66213fe
  Built:            Mon Jun 22 15:45:28 2020
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.2.13
  GitCommit:        7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc:
  Version:          1.0.0-rc10
  GitCommit:
 docker-init:
  Version:          0.18.0
  GitCommit:        fec3683

I follow the official documentation Container runtimes, but I also found the repo in GitHub which describes a bit different the configuration CRI-O (GitHub).

I tried installing cri-o from source but also from the rpm. Both times the result is the same:

Jun 25 13:31:19 hostname kubelet[23665]: I0625 13:31:19.700722   23665 server.go:417] Version: v1.18.2
Jun 25 13:31:19 hostname kubelet[23665]: I0625 13:31:19.701175   23665 plugins.go:100] No cloud provider specified.
Jun 25 13:31:19 hostname kubelet[23665]: I0625 13:31:19.701208   23665 server.go:837] Client rotation is on, will bootstrap in background
Jun 25 13:31:19 hostname kubelet[23665]: F0625 13:31:19.701323   23665 server.go:274] failed to run Kubelet: unable to load bootstrap kubeconfig: stat /etc/kubernetes/bootstrap-kubelet.conf: no such file or directory
Jun 25 13:31:19 hostname systemd[1]: kubelet.service: main process exited, code=exited, status=255/n/a
Jun 25 13:31:19 hostname systemd[1]: Unit kubelet.service entered failed state.
Jun 25 13:31:19 hostname systemd[1]: kubelet.service failed.

From the little that I know if I remember correctly this file /etc/kubernetes/bootstrap-kubelet.conf is autogenerated when kubeadm is started.

Configurations that I have applied.

10-kubeadm.conf:

# cat /etc/systemd/system/kubelet.service.d/10-kubeadm.conf
[Service]
Environment="KUBELET_KUBECONFIG_ARGS=--bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf"
Environment="KUBELET_CONFIG_ARGS=--config=/var/lib/kubelet/config.yaml"
# This is a file that "kubeadm init" and "kubeadm join" generate at runtime, populating the KUBELET_KUBEADM_ARGS variable dynamically
EnvironmentFile=-/var/lib/kubelet/kubeadm-flags.env
# This is a file that the user can use for overrides of the kubelet args as a last resort. Preferably,
# the user should use the .NodeRegistration.KubeletExtraArgs object in the configuration files instead.
# KUBELET_EXTRA_ARGS should be sourced from this file.
EnvironmentFile=-/etc/default/kubelet
ExecStart=
ExecStart=/usr/bin/kubelet $KUBELET_KUBECONFIG_ARGS $KUBELET_CONFIG_ARGS $KUBELET_KUBEADM_ARGS $KUBELET_EXTRA_ARGS

01-log-level.conf:

# cat /etc/crio/crio.conf.d/01-log-level.conf
[crio.runtime]
log_level = "info"

01-cgroup-manager.conf:

# cat /etc/crio/crio.conf.d/01-cgroup-manager.conf
[crio.runtime]
cgroup_manager = "systemd"

kubelet:

# cat /etc/default/kubelet
KUBELET_EXTRA_ARGS=--feature-gates="AllAlpha=false,RunAsGroup=true" --container-runtime=remote --cgroup-driver=systemd --container-runtime-endpoint='unix:///var/run/crio/crio.sock' --runtime-request-timeout=5m

I can verify that the cri-o socket is working as I can pull the images from my repo:

# kubeadm config images pull --image-repository=my.private.repo --kubernetes-version=v1.18.2 --cri-socket unix:///var/run/crio/crio.sock
W0625 13:53:17.554897   29936 configset.go:202] WARNING: kubeadm cannot validate component configs for API groups [kubelet.config.k8s.io kubeproxy.config.k8s.io]
[config/images] Pulled my.private.repo/kube-apiserver:v1.18.2
[config/images] Pulled my.private.repo/kube-controller-manager:v1.18.2
[config/images] Pulled my.private.repo/kube-scheduler:v1.18.2
[config/images] Pulled my.private.repo/kube-proxy:v1.18.2
[config/images] Pulled my.private.repo/pause:3.2
[config/images] Pulled my.private.repo/etcd:3.4.3-0
[config/images] Pulled my.private.repo/coredns:1.6.7

I have spend 3 days and I am not able to figure it out. Can someone with more experience provide more info?

Update: adding init command:

kubeadm init \
        --upload-certs \
        --cri-socket=unix:///var/run/crio/crio.sock \ # /var/run/dockershim.sock 
        --node-name=master-prime \
        --image-repository=my.private.repo \
        --pod-network-cidr=10.96.0.0/16 \
        --kubernetes-version=v1.18.2 \
        --control-plane-endpoint=IP:PORT \
        --apiserver-cert-extra-sans=IP \
        --apiserver-advertise-address=IP

Did you create those kubelet configs by hand, or were the written by [`kubeadm join`](https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-join/#join-workflow)? Because that's the step which emits the `bootstrap-kubelet.conf`, containing the join-token as authentication, which is then used to issue `kubelet` long-term credentials — mdaniel, Jun 25 '20 at 15:52
I created the config files I created them by hand. At least this is what I thought I should do based on documentation no? — Thanos, Jun 26 '20 at 11:08
is the rest of that sentence "and then I ran `kubeadm join`", because that's what I'm trying to get at -- whether you created those config files and expected the Node to spring to life, OR you created those config files, then ran kubeadm join, and things are still not working — mdaniel, Jun 26 '20 at 16:04
I have implemented both solutions. Either with a configuration file or using the flags. I don’t have access to my laptop but I will update my question tomorrow with a sample. This the master primary node in the cluster. And it fails to be launched with cri-o. It is not a master secondary node that it is joining the cluster. — Thanos, Jun 28 '20 at 07:28
Could you take a look at this [tutorial](https://kubevirt.io/2019/KubeVirt_k8s_crio_from_scratch.html) about configurating cri-o and follow up [tutorial](https://kubevirt.io/2019/KubeVirt_k8s_crio_from_scratch_installing_kubernetes.html) for installation? Maybe you find something useful here. Additionally what CNI do you use? Have you configured your cidr in CRI-O configuration file? — Jakub, Jun 30 '20 at 08:53
Still the error persist no matter what I do. It might be a bug I have raised a ticket on the crio team. I will update as soon as I have more information. — Thanos, Jun 30 '20 at 19:12

score 0 · Accepted Answer · answered Dec 02 '20 at 10:20

0

It has been some time that I raised this question and I never answered it. I completely forgot.

The problem with me was that I am launching the cluster on offline cluster.

I managed to figured it out and the CRI-O team asked me to documented in case that other would try to do the same thing.

The full configuration and steps can be found in the official GitHub page: Running kubeadm in an off line network

Hope that this helps someone else in future.

answered Dec 02 '20 at 10:20

Thanos

523
1
7
12

Thanks , --crio-socket supported by the kubeadm from command line or not? – Jepsenwan Jun 05 '21 at 02:06
I am not sure if I understand your question 100%. If you mean through `kubeadm` command line like `kubeadm init` then yes it does. Official documentation [kubeadm init](https://kubernetes.io/docs/reference/setup-tools/kubeadm/kubeadm-init/). – Thanos Jun 10 '21 at 06:44
Thanks. If I don't pickup the runtime using --crio-socket, when there is docker and crio socket coexisting in the same node, who will be picked up as runtime? – Jepsenwan Jun 14 '21 at 02:37
It want start :). It will complain the kubeadm that detects multiple sockets and you need to choose one with the flag that you provided above `--crio-socket`. – Thanos Jun 14 '21 at 07:00

How to setup cri-o with kubeadm and kubelet on Kubernetes 1.18.2?

1 Answers1