I have a one-node cluster deployed with kubespray. I tried 2.14 - the same thing. The cluster successfully starts after deployment, and works for a long time. When I restart the machine all the docker container (apiserver, scheduler, cni, calico, etc.), except etcd, are not working. Systemd kubelet service is up, but its log shows that it is not healthy.
Sep 25 16:07:09 node1 systemd[1]: Started Kubernetes Kubelet Server.
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.378828 4273 flags.go:33] FLAG: --add-dir-header="false"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.378989 4273 flags.go:33] FLAG: --address="0.0.0.0"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.379003 4273 flags.go:33] FLAG: --allowed-unsafe-sysctls="[]"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.379009 4273 flags.go:33] FLAG: --alsologtostderr="false"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.379014 4273 flags.go:33] FLAG: --anonymous-auth="true"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.379019 4273 flags.go:33] FLAG: --application-metrics-count-limit="100"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.379023 4273 flags.go:33] FLAG: --authentication-token-webhook="false"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.379027 4273 flags.go:33] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.379032 4273 flags.go:33] FLAG: --authorization-mode="AlwaysAllow"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.379698 4273 flags.go:33] FLAG: --volume-stats-agg-period="1m0s"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.379734 4273 feature_gate.go:243] feature gates: &{map[]}
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.381339 4273 feature_gate.go:243] feature gates: &{map[]}
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.381391 4273 feature_gate.go:243] feature gates: &{map[]}
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.391492 4273 mount_linux.go:168] Detected OS with systemd
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.391633 4273 server.go:416] Version: v1.17.12
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.391728 4273 feature_gate.go:243] feature gates: &{map[]}
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.391804 4273 feature_gate.go:243] feature gates: &{map[]}
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.391934 4273 plugins.go:100] No cloud provider specified.
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.391956 4273 server.go:532] No cloud provider specified: "" from the config file: ""
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.391968 4273 server.go:822] Client rotation is on, will bootstrap in background
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.393945 4273 bootstrap.go:84] Current kubeconfig file contents are still valid, no bootstrap necessary
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.394040 4273 certificate_store.go:129] Loading cert/key pair from "/var/lib/kubelet/pki/kubelet-client-current.pem".
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.394294 4273 server.go:849] Starting client certificate rotation.
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.394311 4273 certificate_manager.go:275] Certificate rotation is enabled.
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.394462 4273 certificate_manager.go:531] Certificate expiration is 2021-09-25 15:58:29 +0000 UTC, rotation deadline is 2021-08-09 11:53:23.825031508 +0000 UTC
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.394497 4273 certificate_manager.go:281] Waiting 7627h46m14.430536992s for next certificate rotation
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.394860 4273 dynamic_cafile_content.go:166] Starting client-ca-bundle::/etc/kubernetes/ssl/ca.crt
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.395055 4273 manager.go:146] cAdvisor running in container: "/sys/fs/cgroup/cpu,cpuacct/system.slice/kubelet.service"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.414349 4273 fs.go:125] Filesystem UUIDs: map[2020-09-25-15-46-36-00:/dev/sr0 55122f25-8946-4857-952e-48dc1647b4e1:/dev/vda1 88DB-6C04:/dev/vda15]
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.414375 4273 fs.go:126] Filesystem partitions: map[/dev/shm:{mountpoint:/dev/shm major:0 minor:25 fsType:tmpfs blockSize:0} /dev/vda1:{mountpoint:/ major:252 minor:1 fsType:ext4 blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /run/lock:{mountpoint:/run/lock major:0 minor:26 fsType:tmpfs blockSize:0} /run/user/1000:{mountpoint:/run/user/1000 major:0 minor:61 fsType:tmpfs blockSize:0} /sys/fs/cgroup:{mountpoint:/sys/fs/cgroup major:0 minor:27 fsType:tmpfs blockSize:0}]
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.418995 4273 manager.go:193] Machine: {NumCores:8 CpuFrequency:2494136 MemoryCapacity:50641403904 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:6f481a3404734917921596637175f7df SystemUUID:6F481A34-0473-4917-9215-96637175F7DF BootID:4499a41b-6deb-4cf6-906f-a88d6794e657 Filesystems:[{Device:/sys/fs/cgroup DeviceMajor:0 DeviceMinor:27 Capacity:25320701952 Type:vfs Inodes:6181812 HasInodes:true} {Device:/run/user/1000 DeviceMajor:0 DeviceMinor:61 Capacity:5064138752 Type:vfs Inodes:6181812 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:5064142848 Type:vfs Inodes:6181812 HasInodes:true} {Device:/dev/vda1 DeviceMajor:252 DeviceMinor:1 Capacity:103880232960 Type:vfs Inodes:12902400 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:25 Capacity:25320701952 Type:vfs Inodes:6181812 HasInodes:true} {Device:/run/lock DeviceMajor:0 DeviceMinor:26 Capacity:5242880 Type:vfs Inodes:6181812 HasInodes:true}] DiskMap:map[252:0:{Name:vda Major:252 Minor:0 Size:107374182400 Scheduler:none}] NetworkDevices:[{Name:ens3 MacAddress:fa:16:3e:26:40:ce Speed:-1 Mtu:1500}] Topology:[{Id:0 Memory:50641403904 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[{Size:16777216 Type:Unified Level:3}]} {Id:1 Memory:0 HugePages:[] Cores:[{Id:0 Threads:[1] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[{Size:16777216 Type:Unified Level:3}]} {Id:2 Memory:0 HugePages:[] Cores:[{Id:0 Threads:[2] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[{Size:16777216 Type:Unified Level:3}]} {Id:3 Memory:0 HugePages:[] Cores:[{Id:0 Threads:[3] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[{Size:16777216 Type:Unified Level:3}]} {Id:4 Memory:0 HugePages:[] Cores:[{Id:0 Threads:[4] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[{Size:16777216 Type:Unified Level:3}]} {Id:5 Memory:0 HugePages:[] Cores:[{Id:0 Threads:[5] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[{Size:16777216 Type:Unified Level:3}]} {Id:6 Memory:0 HugePages:[] Cores:[{Id:0 Threads:[6] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[{Size:16777216 Type:Unified Level:3}]} {Id:7 Memory:0 HugePages:[] Cores:[{Id:0 Threads:[7] Caches:[{Size:32768 Type:Data Level:1} {Size:32768 Type:Instruction Level:1} {Size:4194304 Type:Unified Level:2}]}] Caches:[{Size:16777216 Type:Unified Level:3}]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.420041 4273 manager.go:199] Version: {KernelVersion:4.15.0-118-generic ContainerOsVersion:Ubuntu 18.04.5 LTS DockerVersion:18.09.7 DockerAPIVersion:1.39 CadvisorVersion: CadvisorRevision:}
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.420193 4273 server.go:642] --cgroups-per-qos enabled, but --cgroup-root was not specified. defaulting to /
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.420530 4273 container_manager_linux.go:265] container manager verified user specified cgroup-root exists: []
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.420548 4273 container_manager_linux.go:270] Creating Container Manager object based on Node Config: {RuntimeCgroupsName:/systemd/system.slice SystemCgroupsName: KubeletCgroupsName:/systemd/system.slice ContainerRuntime:docker CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:cgroupfs KubeletRootDir:/var/lib/kubelet ProtectKernelDefaults:false NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: ReservedSystemCPUs: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[cpu:{i:{value:200 scale:-3} d:{Dec:<nil>} s:200m Format:DecimalSI} memory:{i:{value:512 scale:6} d:{Dec:<nil>} s:512M Format:DecimalSI}] SystemReserved:map[] HardEvictionThresholds:[{Signal:imagefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.15} GracePeriod:0s MinReclaim:<nil>} {Signal:memory.available Operator:LessThan Value:{Quantity:100Mi Percentage:0} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.1} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.inodesFree Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>}]} QOSReserved:map[] ExperimentalCPUManagerPolicy:none ExperimentalCPUManagerReconcilePeriod:10s ExperimentalPodPidsLimit:-1 EnforceCPULimits:true CPUCFSQuotaPeriod:100ms ExperimentalTopologyManagerPolicy:none}
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.420695 4273 fake_topology_manager.go:29] [fake topologymanager] NewFakeManager
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.420702 4273 container_manager_linux.go:305] Creating device plugin manager: true
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.420712 4273 manager.go:126] Creating Device Plugin manager at /var/lib/kubelet/device-plugins/kubelet.sock
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.420731 4273 fake_topology_manager.go:39] [fake topologymanager] AddHintProvider HintProvider: &{kubelet.sock /var/lib/kubelet/device-plugins/ map[] {0 0} <nil> {{} [0 0 0]} 0x1b1eed0 0x6eb9ef8 0x1b1f7a0 map[] map[] map[] map[] map[] 0xc000a6d7d0 [0] 0x6eb9ef8}
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.420763 4273 state_mem.go:36] [cpumanager] initializing new in-memory state store
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.420837 4273 state_mem.go:84] [cpumanager] updated default cpuset: ""
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.420843 4273 state_mem.go:92] [cpumanager] updated cpuset assignments: "map[]"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.420851 4273 state_checkpoint.go:101] [cpumanager] state checkpoint: restored state from checkpoint
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.420855 4273 state_checkpoint.go:102] [cpumanager] state checkpoint: defaultCPUSet:
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.420862 4273 fake_topology_manager.go:39] [fake topologymanager] AddHintProvider HintProvider: &{{0 0} 0x6eb9ef8 10000000000 0xc00049ea20 <nil> <nil> <nil> <nil> map[cpu:{{200 -3} {<nil>} DecimalSI} memory:{{616857600 0} {<nil>} DecimalSI}] 0x6eb9ef8}
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.420925 4273 server.go:1056] Using root directory: /var/lib/kubelet
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.420947 4273 kubelet.go:286] Adding pod path: /etc/kubernetes/manifests
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.420974 4273 file.go:68] Watching path "/etc/kubernetes/manifests"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.420987 4273 kubelet.go:311] Watching apiserver
Sep 25 16:07:09 node1 kubelet[4273]: E0925 16:07:09.421729 4273 reflector.go:153] k8s.io/kubernetes/pkg/kubelet/kubelet.go:449: Failed to list *v1.Service: Get https://10.34.12.170:6443/api/v1/services?limit=500&resourceVersion=0: dial tcp 10.34.12.170:6443: connect: connection refused
Sep 25 16:07:09 node1&limit=500&resourceVersion=0: dial tcp 10.34.12.170:6443: connect: connection refused
Sep 25 16:07:09 node1&limit=500&resourceVersion=0: dial tcp 10.34.12.170:6443: connect: connection refused
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.424494 4273 client.go:75] Connecting to docker on unix:///var/run/docker.sock
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.424517 4273 client.go:104] Start docker client with request timeout=2m0s
Sep 25 16:07:09 node1 kubelet[4273]: W0925 16:07:09.425742 4273 docker_service.go:564] Hairpin mode set to "promiscuous-bridge" but kubenet is not enabled, falling back to "hairpin-veth"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.425764 4273 docker_service.go:240] Hairpin mode set to "hairpin-veth"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.459931 4273 plugins.go:166] Loaded network plugin "cni"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.459991 4273 docker_service.go:255] Docker cri networking managed by cni
Sep 25 16:07:09 node1 Labels:[] ExperimentalBuild:false ServerVersion:18.09.7 ClusterStore: ClusterAdvertise: Runtimes:map[runc:{Path:runc Args:[]}] DefaultRuntime:runc Swarm:{NodeID: NodeAddr: LocalNodeState:inactive ControlAvailable:false Error: RemoteManagers:[] Nodes:0 Managers:0 Cluster:<nil> Warnings:[]} LiveRestoreEnabled:false Isolation: InitBinary:docker-init ContainerdCommit:{ID:8fba4e9a7d01810a393d5d25a3621dc101981175 Expected:8fba4e9a7d01810a393d5d25a3621dc101981175} RuncCommit:{ID:dc9208a3303feef5b3839f4323d9beb36df0a9dd Expected:dc9208a3303feef5b3839f4323d9beb36df0a9dd} InitCommit:{ID:fec3683 Expected:fec3683} SecurityOptions:[name=apparmor name=seccomp,profile=default] ProductLicense:Community Engine Warnings:[WARNING: No swap limit support]}
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.468858 4273 docker_service.go:273] Setting cgroupDriver to cgroupfs
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.468983 4273 kubelet.go:642] Starting the GRPC server for the docker CRI shim.
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.469073 4273 container_manager_linux.go:118] Configure resource-only container "/systemd/system.slice" with memory limit: 35448982732
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.469091 4273 docker_server.go:59] Start dockershim grpc server
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.478669 4273 remote_runtime.go:59] parsed scheme: ""
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.478689 4273 remote_runtime.go:59] scheme "" not registered, fallback to default scheme
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.478714 4273 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{/var/run/dockershim.sock 0 <nil>}] <nil>}
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.478722 4273 clientconn.go:577] ClientConn switching balancer to "pick_first"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.478749 4273 remote_image.go:50] parsed scheme: ""
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.478755 4273 remote_image.go:50] scheme "" not registered, fallback to default scheme
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.478764 4273 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{/var/run/dockershim.sock 0 <nil>}] <nil>}
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.478769 4273 clientconn.go:577] ClientConn switching balancer to "pick_first"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.478889 4273 balancer_conn_wrappers.go:127] pickfirstBalancer: HandleSubConnStateChange: 0xc00026e020, CONNECTING
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.478928 4273 balancer_conn_wrappers.go:127] pickfirstBalancer: HandleSubConnStateChange: 0xc000a42020, CONNECTING
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.479399 4273 balancer_conn_wrappers.go:127] pickfirstBalancer: HandleSubConnStateChange: 0xc00026e020, READY
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.479426 4273 balancer_conn_wrappers.go:127] pickfirstBalancer: HandleSubConnStateChange: 0xc000a42020, READY
Sep 25 16:07:09 node1 kubelet[4273]: E0925 16:07:09.870533 4273 aws_credentials.go:77] while getting AWS credentials NoCredentialProviders: no valid providers in chain. Deprecated.
Sep 25 16:07:09 node1 kubelet[4273]: For verbose messaging see aws.Config.CredentialsChainVerboseErrors
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.872026 4273 kuberuntime_manager.go:217] Container runtime docker initialized, version: 18.09.7, apiVersion: 1.39.0
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.872387 4273 plugins.go:631] Loaded volume plugin "kubernetes.io/aws-ebs"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.872485 4273 plugins.go:631] Loaded volume plugin "kubernetes.io/gce-pd"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.872550 4273 plugins.go:631] Loaded volume plugin "kubernetes.io/cinder"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.872602 4273 plugins.go:631] Loaded volume plugin "kubernetes.io/azure-disk"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.872651 4273 plugins.go:631] Loaded volume plugin "kubernetes.io/azure-file"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.872701 4273 plugins.go:631] Loaded volume plugin "kubernetes.io/vsphere-volume"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.872763 4273 plugins.go:631] Loaded volume plugin "kubernetes.io/empty-dir"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.872813 4273 plugins.go:631] Loaded volume plugin "kubernetes.io/git-repo"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.872872 4273 plugins.go:631] Loaded volume plugin "kubernetes.io/host-path"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.872926 4273 plugins.go:631] Loaded volume plugin "kubernetes.io/nfs"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.872976 4273 plugins.go:631] Loaded volume plugin "kubernetes.io/secret"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.873041 4273 plugins.go:631] Loaded volume plugin "kubernetes.io/iscsi"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.873091 4273 plugins.go:631] Loaded volume plugin "kubernetes.io/glusterfs"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.873151 4273 plugins.go:631] Loaded volume plugin "kubernetes.io/rbd"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.873200 4273 plugins.go:631] Loaded volume plugin "kubernetes.io/quobyte"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.873251 4273 plugins.go:631] Loaded volume plugin "kubernetes.io/cephfs"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.873298 4273 plugins.go:631] Loaded volume plugin "kubernetes.io/downward-api"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.873347 4273 plugins.go:631] Loaded volume plugin "kubernetes.io/fc"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.873397 4273 plugins.go:631] Loaded volume plugin "kubernetes.io/flocker"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.873445 4273 plugins.go:631] Loaded volume plugin "kubernetes.io/configmap"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.873496 4273 plugins.go:631] Loaded volume plugin "kubernetes.io/projected"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.873556 4273 plugins.go:631] Loaded volume plugin "kubernetes.io/portworx-volume"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.873623 4273 plugins.go:631] Loaded volume plugin "kubernetes.io/scaleio"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.873681 4273 plugins.go:631] Loaded volume plugin "kubernetes.io/local-volume"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.873729 4273 plugins.go:631] Loaded volume plugin "kubernetes.io/storageos"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.873816 4273 plugins.go:631] Loaded volume plugin "kubernetes.io/csi"
Sep 25 16:07:09 node1: dial tcp 10.34.12.170:6443: connect: connection refused
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.875105 4273 server.go:1114] Started kubelet
Sep 25 16:07:09 node1 kubelet[4273]: E0925 16:07:09.875145 4273 kubelet.go:1305] Image garbage collection failed once. Stats initialization may not have completed yet: failed to get imageFs info: unable to find data in memory cache
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.875186 4273 server.go:144] Starting to listen on 10.34.12.170:10250
Sep 25 16:07:09 node1 kubelet[4273]: E0925 16:07:09.875457 4273 event.go:272] Unable to write event: 'Post https://10.34.12.170:6443/api/v1/namespaces/default/events: dial tcp 10.34.12.170:6443: connect: connection refused' (may retry after sleeping)
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.876063 4273 fs_resource_analyzer.go:64] Starting FS ResourceAnalyzer
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.876153 4273 volume_manager.go:263] The desired_state_of_world populator starts
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.876163 4273 volume_manager.go:265] Starting Kubelet Volume Manager
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.876550 4273 desired_state_of_world_populator.go:138] Desired state populator starts to run
Sep 25 16:07:09 node1 kubelet[4273]: E0925 16:07:09.876729 4273 reflector.go:153] k8s.io/client-go/informers/factory.go:135: Failed to list *v1beta1.CSIDriver: Get https://10.34.12.170:6443/apis/storage.k8s.io/v1beta1/csidrivers?limit=500&resourceVersion=0: dial tcp 10.34.12.170:6443: connect: connection refused
Sep 25 16:07:09 node1?timeout=10s: dial tcp 10.34.12.170:6443: connect: connection refused
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.877999 4273 server.go:384] Adding debug handlers to kubelet server.
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.891555 4273 factory.go:356] Registering Docker factory
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.891578 4273 factory.go:54] Registering systemd factory
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.891705 4273 clientconn.go:104] parsed scheme: "unix"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.891714 4273 clientconn.go:104] scheme "unix" not registered, fallback to default scheme
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.891735 4273 passthrough.go:48] ccResolverWrapper: sending update to cc: {[{unix:///run/containerd/containerd.sock 0 <nil>}] <nil>}
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.891743 4273 clientconn.go:577] ClientConn switching balancer to "pick_first"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.891797 4273 balancer_conn_wrappers.go:127] pickfirstBalancer: HandleSubConnStateChange: 0xc00037cd00, CONNECTING
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.892584 4273 balancer_conn_wrappers.go:127] pickfirstBalancer: HandleSubConnStateChange: 0xc00037cd00, READY
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.893173 4273 factory.go:137] Registering containerd factory
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.893533 4273 factory.go:101] Registering Raw factory
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.893662 4273 manager.go:1158] Started watching for new ooms in manager
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.894666 4273 manager.go:272] Starting recovery of all containers
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.897183 4273 kubelet_network_linux.go:150] Not using `--random-fully` in the MASQUERADE rule for iptables because the local version of iptables does not support it
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.898224 4273 status_manager.go:157] Starting to sync pod status with apiserver
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.898331 4273 kubelet.go:1823] Starting kubelet main sync loop.
Sep 25 16:07:09 node1 kubelet[4273]: E0925 16:07:09.898413 4273 kubelet.go:1847] skipping pod synchronization - [container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]
Sep 25 16:07:09 node1 kubelet[4273]: E0925 16:07:09.899415 4273 reflector.go:153] k8s.io/client-go/informers/factory.go:135: Failed to list *v1beta1.RuntimeClass: Get https://10.34.12.170:6443/apis/node.k8s.io/v1beta1/runtimeclasses?limit=500&resourceVersion=0: dial tcp 10.34.12.170:6443: connect: connection refused
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.904467 4273 manager.go:277] Recovery completed
Sep 25 16:07:09 node1 kubelet[4273]: W0925 16:07:09.923594 4273 docker_sandbox.go:394] failed to read pod IP from plugin/docker: networkPlugin cni failed on the status hook for pod "kubernetes-metrics-scraper-747b4fd5cd-f2gtr_kube-system": CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container "15b9965fb59f8b9f590ff1f01c7bf8c71746da8dcb0163a7277b7d1664f09b52"
Sep 25 16:07:09 node1 kubelet[4273]: W0925 16:07:09.928100 4273 docker_sandbox.go:394] failed to read pod IP from plugin/docker: networkPlugin cni failed on the status hook for pod "kubernetes-dashboard-77475cf576-xc6br_kube-system": CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container "3652b305aec8a60b50602b5ca12e9335a8e35c270cbcfcb7896139de97c73ace"
Sep 25 16:07:09 node1 kubelet[4273]: W0925 16:07:09.944457 4273 docker_sandbox.go:394] failed to read pod IP from plugin/docker: networkPlugin cni failed on the status hook for pod "dns-autoscaler-85f898cd5c-jv6cb_kube-system": CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container "55ee03cc56e650563ab445bccab7f327c7b8c66e086efbeb7d93bfb816aba838"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.947817 4273 kubelet_node_status.go:294] Setting node annotation to enable volume controller attach/detach
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.947980 4273 setters.go:73] Using node IP: "10.34.12.170"
Sep 25 16:07:09 node1
Sep 25 16:07:09 node1
Sep 25 16:07:09 node1
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.949354 4273 cpu_manager.go:173] [cpumanager] starting with none policy
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.949359 4273 cpu_manager.go:174] [cpumanager] reconciling every 10s
Sep 25 16:07:09 node1 kubelet[4273]: W0925 16:07:09.949360 4273 docker_sandbox.go:394] failed to read pod IP from plugin/docker: networkPlugin cni failed on the status hook for pod "coredns-76798d84dd-6bmth_kube-system": CNI failed to retrieve network namespace path: cannot find network namespace for the terminated container "bde82b4692238cdcc8bb0860d9773ddbf57fc09d59388f655757224ef463ff1a"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.949365 4273 policy_none.go:43] [cpumanager] none policy: Start
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.950266 4273 manager.go:226] Starting Device Plugin manager
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.950956 4273 manager.go:268] Serving device plugin registration server on "/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.951022 4273 plugin_watcher.go:54] Plugin Watcher Start at /var/lib/kubelet/plugins_registry
Sep 25 16:07:09 node1 kubelet[4273]: I0925 16:07:09.951107 4273 plugin_manager.go:112] The desired_state_of_world populator (plugin watcher) starts
Basically, it tires to talk to apiserver unsuccessfully.
What I have observed and become very confused that neither apiserver container nor cni/calico container (docker ps -a) never get get started. Is not it kubelet suppose to start those, if so, why it attemps to talk to api server without trying to start it? If no kubelete then what is starting the apiserver, the restart policy on the api server container is no.
