I had restarted the server (master node) and I get since then the following message when I want to use kubelet:
The connection to the server YYY.YYY.YYY.YY:6443 was refused - did you specify the right host or port?
In the log of kubelet it says “Error getting node” err=“node "jupyterhub-test" not found”. jupyterhub-test is the master node.
The only things I always found while researching, I already tested.
- restart kubelet
- restart docker
- swapoff -a
I can’t figure out from the log of kubelet where exactly the problem is and I hope you can help me.
Cluster information:
Kubernetes version: Client Version: version.Info{Major:“1”, Minor:“24”, GitVersion:“v1.24.1”, GitCommit:“3ddd0f45aa91e2f30c70734b175631bec5b5825a”, GitTreeState:“clean”, BuildDate:“2022-05-24T12:26:19Z”, GoVersion:“go1.18.2”, Compiler:“gc”, Platform:“linux/amd64”}
Installation method: kubeadm init
Host OS: Ubuntu Ubuntu 20.04.4 LTS
kubelet log:
Sep 14 08:20:37 jupyterhub-test systemd[1]: Started kubelet: The Kubernetes Node Agent.
-- Subject: A start job for unit kubelet.service has finished successfully
-- Defined-By: systemd
-- Support: http://www.ubuntu.com/support
--
-- A start job for unit kubelet.service has finished successfully.
--
-- The job identifier is 7916.
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: Flag --container-runtime has been deprecated, will be removed in 1.27 as the only valid value is 'remote'
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.27. Image garbage collector will get sandbox image information from CRI.
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.562908 178160 server.go:193] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: Flag --container-runtime has been deprecated, will be removed in 1.27 as the only valid value is 'remote'
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.27. Image garbage collector will get sandbox image information from CRI.
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.571728 178160 server.go:399] "Kubelet version" kubeletVersion="v1.24.1"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.571751 178160 server.go:401] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.571927 178160 server.go:813] "Client rotation is on, will bootstrap in background"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.572804 178160 certificate_store.go:130] Loading cert/key pair from "/var/lib/kubelet/pki/kubelet-client-current.pem".
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.573378 178160 dynamic_cafile_content.go:157] "Starting controller" name="client-ca-bundle::/etc/kubernetes/pki/ca.crt"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.575898 178160 server.go:648] "--cgroups-per-qos enabled, but --cgroup-root was not specified. defaulting to /"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.576200 178160 container_manager_linux.go:262] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.576261 178160 container_manager_linux.go:267] "Creating Container Manager object based on Node Config" nodeConfig={RuntimeCgroupsName: SystemCgroupsName: KubeletCgroupsName: KubeletOOMScoreAdj:-999 ContainerRuntime: CgroupsPerQOS:true CgroupRoot:/ CgroupDriver:systemd KubeletRootDir:/var/lib/kubelet ProtectKernelDefaults:false NodeAllocatableConfig:{KubeReservedCgroupName: SystemReservedCgroupName: ReservedSystemCPUs: EnforceNodeAllocatable:map[pods:{}] KubeReserved:map[] SystemReserved:map[] HardEvictionThresholds:[{Signal:imagefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.15} GracePeriod:0s MinReclaim:<nil>} {Signal:memory.available Operator:LessThan Value:{Quantity:100Mi Percentage:0} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.available Operator:LessThan Value:{Quantity:<nil> Percentage:0.1} GracePeriod:0s MinReclaim:<nil>} {Signal:nodefs.inodesFree Operator:LessThan Value:{Quantity:<nil> Percentage:0.05} GracePeriod:0s MinReclaim:<nil>}]} QOSReserved:map[] ExperimentalCPUManagerPolicy:none ExperimentalCPUManagerPolicyOptions:map[] ExperimentalTopologyManagerScope:container ExperimentalCPUManagerReconcilePeriod:10s ExperimentalMemoryManagerPolicy:None ExperimentalMemoryManagerReservedMemory:[] ExperimentalPodPidsLimit:-1 EnforceCPULimits:true CPUCFSQuotaPeriod:100ms ExperimentalTopologyManagerPolicy:none}
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.576319 178160 topology_manager.go:133] "Creating topology manager with policy per scope" topologyPolicyName="none" topologyScopeName="container"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.576328 178160 container_manager_linux.go:302] "Creating device plugin manager" devicePluginEnabled=true
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.576359 178160 state_mem.go:36] "Initialized new in-memory state store"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.579169 178160 kubelet.go:376] "Attempting to sync node with API server"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.579187 178160 kubelet.go:267] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.579203 178160 kubelet.go:278] "Adding apiserver pod source"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.579211 178160 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: WARN 08:20:37.580088 178160 reflector.go:324] vendor/k8s.io/client-go/informers/factory.go:134: failed to list *v1.Node: Get "https://YYY.YYY.YYY.YY:6443/api/v1/nodes?fieldSelector=metadata.name%3Djupyterhub-test&limit=500&resourceVersion=0": dial tcp YYY.YYY.YYY.YY:6443: connect: connection refused
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: ERROR 08:20:37.580150 178160 reflector.go:138] vendor/k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Node: failed to list *v1.Node: Get "https://YYY.YYY.YYY.YY:6443/api/v1/nodes?fieldSelector=metadata.name%3Djupyterhub-test&limit=500&resourceVersion=0": dial tcp YYY.YYY.YYY.YY:6443: connect: connection refused
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.580230 178160 kuberuntime_manager.go:239] "Container runtime initialized" containerRuntime="containerd" version="1.6.6" apiVersion="v1"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.581022 178160 server.go:1181] "Started kubelet"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.581694 178160 server.go:150] "Starting to listen" address="0.0.0.0" port=10250
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: ERROR 08:20:37.582261 178160 cri_stats_provider.go:455] "Failed to get the info of the filesystem with mountpoint" err="unable to find data in memory cache" mountpoint="/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.582351 178160 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: ERROR 08:20:37.582738 178160 event.go:276] Unable to write event: '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"jupyterhub-test.1714ac53a07755b3", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ZZZ_DeprecatedClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Node", Namespace:"", Name:"jupyterhub-test", UID:"jupyterhub-test", APIVersion:"", ResourceVersion:"", FieldPath:""}, Reason:"Starting", Message:"Starting kubelet.", Source:v1.EventSource{Component:"kubelet", Host:"jupyterhub-test"}, FirstTimestamp:time.Date(2022, time.September, 14, 8, 20, 37, 580993971, time.Local), LastTimestamp:time.Date(2022, time.September, 14, 8, 20, 37, 580993971, time.Local), Count:1, Type:"Normal", EventTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'Post "https://YYY.YYY.YYY.YY:6443/api/v1/namespaces/default/events": dial tcp YYY.YYY.YYY.YY:6443: connect: connection refused'(may retry after sleeping)
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: WARN 08:20:37.582943 178160 reflector.go:324] vendor/k8s.io/client-go/informers/factory.go:134: failed to list *v1.Service: Get "https://YYY.YYY.YYY.YY:6443/api/v1/services?limit=500&resourceVersion=0": dial tcp YYY.YYY.YYY.YY:6443: connect: connection refused
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: ERROR 08:20:37.583036 178160 reflector.go:138] vendor/k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.Service: failed to list *v1.Service: Get "https://YYY.YYY.YYY.YY:6443/api/v1/services?limit=500&resourceVersion=0": dial tcp YYY.YYY.YYY.YY:6443: connect: connection refused
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.583351 178160 server.go:410] "Adding debug handlers to kubelet server"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: ERROR 08:20:37.582359 178160 kubelet.go:1298] "Image garbage collection failed once. Stats initialization may not have completed yet" err="invalid capacity 0 on image filesystem"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.585879 178160 volume_manager.go:289] "Starting Kubelet Volume Manager"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.585928 178160 desired_state_of_world_populator.go:145] "Desired state populator starts to run"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: ERROR 08:20:37.586167 178160 controller.go:144] failed to ensure lease exists, will retry in 200ms, error: Get "https://YYY.YYY.YYY.YY:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/jupyterhub-test?timeout=10s": dial tcp YYY.YYY.YYY.YY:6443: connect: connection refused
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: WARN 08:20:37.586759 178160 reflector.go:324] vendor/k8s.io/client-go/informers/factory.go:134: failed to list *v1.CSIDriver: Get "https://YYY.YYY.YYY.YY:6443/apis/storage.k8s.io/v1/csidrivers?limit=500&resourceVersion=0": dial tcp YYY.YYY.YYY.YY:6443: connect: connection refused
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: ERROR 08:20:37.586877 178160 reflector.go:138] vendor/k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.CSIDriver: failed to list *v1.CSIDriver: Get "https://YYY.YYY.YYY.YY:6443/apis/storage.k8s.io/v1/csidrivers?limit=500&resourceVersion=0": dial tcp YYY.YYY.YYY.YY:6443: connect: connection refused
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.598425 178160 kubelet_network_linux.go:76] "Initialized protocol iptables rules." protocol=IPv4
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.614860 178160 kubelet_network_linux.go:76] "Initialized protocol iptables rules." protocol=IPv6
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.614881 178160 status_manager.go:161] "Starting to sync pod status with apiserver"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.614893 178160 kubelet.go:1974] "Starting kubelet main sync loop"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: ERROR 08:20:37.615122 178160 kubelet.go:1998] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: WARN 08:20:37.616041 178160 reflector.go:324] vendor/k8s.io/client-go/informers/factory.go:134: failed to list *v1.RuntimeClass: Get "https://YYY.YYY.YYY.YY:6443/apis/node.k8s.io/v1/runtimeclasses?limit=500&resourceVersion=0": dial tcp YYY.YYY.YYY.YY:6443: connect: connection refused
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: ERROR 08:20:37.616074 178160 reflector.go:138] vendor/k8s.io/client-go/informers/factory.go:134: Failed to watch *v1.RuntimeClass: failed to list *v1.RuntimeClass: Get "https://YYY.YYY.YYY.YY:6443/apis/node.k8s.io/v1/runtimeclasses?limit=500&resourceVersion=0": dial tcp YYY.YYY.YYY.YY:6443: connect: connection refused
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.665155 178160 cpu_manager.go:213] "Starting CPU manager" policy="none"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.665168 178160 cpu_manager.go:214] "Reconciling" reconcilePeriod="10s"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.665179 178160 state_mem.go:36] "Initialized new in-memory state store"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.665277 178160 state_mem.go:88] "Updated default CPUSet" cpuSet=""
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.665286 178160 state_mem.go:96] "Updated CPUSet assignments" assignments=map[]
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.665291 178160 policy_none.go:49] "None policy: Start"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.666175 178160 memory_manager.go:168] "Starting memorymanager" policy="None"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.666199 178160 state_mem.go:35] "Initializing new in-memory state store"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.666468 178160 state_mem.go:75] "Updated machine memory state"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.670363 178160 manager.go:610] "Failed to read data from checkpoint" checkpoint="kubelet_internal_checkpoint" err="checkpoint is not found"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.670567 178160 plugin_manager.go:114] "Starting Kubelet Plugin Manager"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: ERROR 08:20:37.671373 178160 eviction_manager.go:254] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"jupyterhub-test\" not found"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: ERROR 08:20:37.686080 178160 kubelet.go:2419] "Error getting node" err="node \"jupyterhub-test\" not found"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.686550 178160 kubelet_node_status.go:70] "Attempting to register node" node="jupyterhub-test"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: ERROR 08:20:37.686984 178160 kubelet_node_status.go:92] "Unable to register node with API server" err="Post \"https://YYY.YYY.YYY.YY:6443/api/v1/nodes\": dial tcp YYY.YYY.YYY.YY:6443: connect: connection refused" node="jupyterhub-test"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.716206 178160 topology_manager.go:200] "Topology Admit Handler"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.716954 178160 topology_manager.go:200] "Topology Admit Handler"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.717589 178160 topology_manager.go:200] "Topology Admit Handler"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.717932 178160 status_manager.go:664] "Failed to get status for pod" podUID=6caff56f64d1ee735407a5a7ba6a787a pod="kube-system/etcd-jupyterhub-test" err="Get \"https://YYY.YYY.YYY.YY:6443/api/v1/namespaces/kube-system/pods/etcd-jupyterhub-test\": dial tcp YYY.YYY.YYY.YY:6443: connect: connection refused"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.718167 178160 topology_manager.go:200] "Topology Admit Handler"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.718644 178160 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="daf5debb6988d121c82972b4d0f6c1935c005063b7f42da0c10773efbf0525e1"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.718776 178160 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="435139d2c5996b2f24600f7cd4911fc77a92bf06e7361347971e70a68e0a54e5"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.718849 178160 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="3554bcaebcaf46d227f480181f505d48af9a6f9d389e93f1c6d3fab703db8df4"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.718927 178160 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="257b423a7d1730e88f277239d535923ffac18d5430b723aee160d50af5417553"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.718995 178160 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="cae22d913fbc0ce98a6e7b0d8b13cef758179416dea8f7a9dd875ab5172d8a4f"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.719055 178160 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="c86abd3d68a6a6f596481f8784db328cc2bf9951b0a514bff979e4243b63e9ce"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.719113 178160 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="0db53f26fd3fc078c02941576d2187b8525fe2da77a2e6dffff797bba8b12213"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.719177 178160 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="19d5c5e57cd43c00235306c69378fd6a9d224a337a79ab0d19417bb7ae8c91b4"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.719239 178160 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="84298f6839000667cb1d02bc812a84c94920e088d5649821fb6ebe3dabc13698"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.719301 178160 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="4d624b46c1721a68bf6b8b1a57d0aa23ee8d664108114337234826a1e053c991"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.719370 178160 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="03454711bc28b88c9a5b4eeb8ab8c962e758bb8ae34cbee798ecd5651ca37bc8"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.719429 178160 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="1a29e83515332ebd4cf4eabecdaa4c305e280de78a90d90210fd4d467a28bffd"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.719490 178160 pod_container_deletor.go:79] "Container not found in pod's containers" containerID="1b8edc5b07c4a5eab18f9e7edb73b089ea303a5f393b650deb550935456c5df8"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.718813 178160 status_manager.go:664] "Failed to get status for pod" podUID=8cdd26801ec71603976b3d4c3c72beae pod="kube-system/kube-apiserver-jupyterhub-test" err="Get \"https://YYY.YYY.YYY.YY:6443/api/v1/namespaces/kube-system/pods/kube-apiserver-jupyterhub-test\": dial tcp YYY.YYY.YYY.YY:6443: connect: connection refused"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.719777 178160 status_manager.go:664] "Failed to get status for pod" podUID=c9c46b502c40273f3cde89de382eb13b pod="kube-system/kube-controller-manager-jupyterhub-test" err="Get \"https://YYY.YYY.YYY.YY:6443/api/v1/namespaces/kube-system/pods/kube-controller-manager-jupyterhub-test\": dial tcp YYY.YYY.YYY.YY:6443: connect: connection refused"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.720022 178160 status_manager.go:664] "Failed to get status for pod" podUID=a7527c69e782fb5d6404c82767da6341 pod="kube-system/kube-scheduler-jupyterhub-test" err="Get \"https://YYY.YYY.YYY.YY:6443/api/v1/namespaces/kube-system/pods/kube-scheduler-jupyterhub-test\": dial tcp YYY.YYY.YYY.YY:6443: connect: connection refused"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: ERROR 08:20:37.786540 178160 controller.go:144] failed to ensure lease exists, will retry in 400ms, error: Get "https://YYY.YYY.YYY.YY:6443/apis/coordination.k8s.io/v1/namespaces/kube-node-lease/leases/jupyterhub-test?timeout=10s": dial tcp YYY.YYY.YYY.YY:6443: connect: connection refused
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: ERROR 08:20:37.786583 178160 kubelet.go:2419] "Error getting node" err="node \"jupyterhub-test\" not found"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.786993 178160 reconciler.go:270] "operationExecutor.VerifyControllerAttachedVolume started for volume \"k8s-certs\" (UniqueName: \"kubernetes.io/host-path/8cdd26801ec71603976b3d4c3c72beae-k8s-certs\") pod \"kube-apiserver-jupyterhub-test\" (UID: \"8cdd26801ec71603976b3d4c3c72beae\") " pod="kube-system/kube-apiserver-jupyterhub-test"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.787171 178160 reconciler.go:270] "operationExecutor.VerifyControllerAttachedVolume started for volume \"usr-local-share-ca-certificates\" (UniqueName: \"kubernetes.io/host-path/8cdd26801ec71603976b3d4c3c72beae-usr-local-share-ca-certificates\") pod \"kube-apiserver-jupyterhub-test\" (UID: \"8cdd26801ec71603976b3d4c3c72beae\") " pod="kube-system/kube-apiserver-jupyterhub-test"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.787275 178160 reconciler.go:270] "operationExecutor.VerifyControllerAttachedVolume started for volume \"usr-share-ca-certificates\" (UniqueName: \"kubernetes.io/host-path/8cdd26801ec71603976b3d4c3c72beae-usr-share-ca-certificates\") pod \"kube-apiserver-jupyterhub-test\" (UID: \"8cdd26801ec71603976b3d4c3c72beae\") " pod="kube-system/kube-apiserver-jupyterhub-test"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.787372 178160 reconciler.go:270] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etcd-certs\" (UniqueName: \"kubernetes.io/host-path/6caff56f64d1ee735407a5a7ba6a787a-etcd-certs\") pod \"etcd-jupyterhub-test\" (UID: \"6caff56f64d1ee735407a5a7ba6a787a\") " pod="kube-system/etcd-jupyterhub-test"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.787406 178160 reconciler.go:270] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etcd-data\" (UniqueName: \"kubernetes.io/host-path/6caff56f64d1ee735407a5a7ba6a787a-etcd-data\") pod \"etcd-jupyterhub-test\" (UID: \"6caff56f64d1ee735407a5a7ba6a787a\") " pod="kube-system/etcd-jupyterhub-test"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.787431 178160 reconciler.go:270] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-certs\" (UniqueName: \"kubernetes.io/host-path/8cdd26801ec71603976b3d4c3c72beae-ca-certs\") pod \"kube-apiserver-jupyterhub-test\" (UID: \"8cdd26801ec71603976b3d4c3c72beae\") " pod="kube-system/kube-apiserver-jupyterhub-test"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.787452 178160 reconciler.go:270] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-ca-certificates\" (UniqueName: \"kubernetes.io/host-path/8cdd26801ec71603976b3d4c3c72beae-etc-ca-certificates\") pod \"kube-apiserver-jupyterhub-test\" (UID: \"8cdd26801ec71603976b3d4c3c72beae\") " pod="kube-system/kube-apiserver-jupyterhub-test"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.787479 178160 reconciler.go:270] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-pki\" (UniqueName: \"kubernetes.io/host-path/8cdd26801ec71603976b3d4c3c72beae-etc-pki\") pod \"kube-apiserver-jupyterhub-test\" (UID: \"8cdd26801ec71603976b3d4c3c72beae\") " pod="kube-system/kube-apiserver-jupyterhub-test"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: ERROR 08:20:37.887309 178160 kubelet.go:2419] "Error getting node" err="node \"jupyterhub-test\" not found"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.887665 178160 kubelet_node_status.go:70] "Attempting to register node" node="jupyterhub-test"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.889786 178160 reconciler.go:270] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-certs\" (UniqueName: \"kubernetes.io/host-path/c9c46b502c40273f3cde89de382eb13b-ca-certs\") pod \"kube-controller-manager-jupyterhub-test\" (UID: \"c9c46b502c40273f3cde89de382eb13b\") " pod="kube-system/kube-controller-manager-jupyterhub-test"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.889824 178160 reconciler.go:270] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-pki\" (UniqueName: \"kubernetes.io/host-path/c9c46b502c40273f3cde89de382eb13b-etc-pki\") pod \"kube-controller-manager-jupyterhub-test\" (UID: \"c9c46b502c40273f3cde89de382eb13b\") " pod="kube-system/kube-controller-manager-jupyterhub-test"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.889845 178160 reconciler.go:270] "operationExecutor.VerifyControllerAttachedVolume started for volume \"k8s-certs\" (UniqueName: \"kubernetes.io/host-path/c9c46b502c40273f3cde89de382eb13b-k8s-certs\") pod \"kube-controller-manager-jupyterhub-test\" (UID: \"c9c46b502c40273f3cde89de382eb13b\") " pod="kube-system/kube-controller-manager-jupyterhub-test"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.889865 178160 reconciler.go:270] "operationExecutor.VerifyControllerAttachedVolume started for volume \"usr-share-ca-certificates\" (UniqueName: \"kubernetes.io/host-path/c9c46b502c40273f3cde89de382eb13b-usr-share-ca-certificates\") pod \"kube-controller-manager-jupyterhub-test\" (UID: \"c9c46b502c40273f3cde89de382eb13b\") " pod="kube-system/kube-controller-manager-jupyterhub-test"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.889885 178160 reconciler.go:270] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubeconfig\" (UniqueName: \"kubernetes.io/host-path/a7527c69e782fb5d6404c82767da6341-kubeconfig\") pod \"kube-scheduler-jupyterhub-test\" (UID: \"a7527c69e782fb5d6404c82767da6341\") " pod="kube-system/kube-scheduler-jupyterhub-test"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.889930 178160 reconciler.go:270] "operationExecutor.VerifyControllerAttachedVolume started for volume \"flexvolume-dir\" (UniqueName: \"kubernetes.io/host-path/c9c46b502c40273f3cde89de382eb13b-flexvolume-dir\") pod \"kube-controller-manager-jupyterhub-test\" (UID: \"c9c46b502c40273f3cde89de382eb13b\") " pod="kube-system/kube-controller-manager-jupyterhub-test"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.889950 178160 reconciler.go:270] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubeconfig\" (UniqueName: \"kubernetes.io/host-path/c9c46b502c40273f3cde89de382eb13b-kubeconfig\") pod \"kube-controller-manager-jupyterhub-test\" (UID: \"c9c46b502c40273f3cde89de382eb13b\") " pod="kube-system/kube-controller-manager-jupyterhub-test"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.889970 178160 reconciler.go:270] "operationExecutor.VerifyControllerAttachedVolume started for volume \"usr-local-share-ca-certificates\" (UniqueName: \"kubernetes.io/host-path/c9c46b502c40273f3cde89de382eb13b-usr-local-share-ca-certificates\") pod \"kube-controller-manager-jupyterhub-test\" (UID: \"c9c46b502c40273f3cde89de382eb13b\") " pod="kube-system/kube-controller-manager-jupyterhub-test"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: INFO 08:20:37.890011 178160 reconciler.go:270] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-ca-certificates\" (UniqueName: \"kubernetes.io/host-path/c9c46b502c40273f3cde89de382eb13b-etc-ca-certificates\") pod \"kube-controller-manager-jupyterhub-test\" (UID: \"c9c46b502c40273f3cde89de382eb13b\") " pod="kube-system/kube-controller-manager-jupyterhub-test"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: ERROR 08:20:37.890469 178160 kubelet_node_status.go:92] "Unable to register node with API server" err="Post \"https://YYY.YYY.YYY.YY:6443/api/v1/nodes\": dial tcp YYY.YYY.YYY.YY:6443: connect: connection refused" node="jupyterhub-test"
Sep 14 08:20:37 jupyterhub-test kubelet[178160]: ERROR 08:20:37.988241 178160 kubelet.go:2419] "Error getting node" err="node \"jupyterhub-test\" not found"
Sep 14 08:20:38 jupyterhub-test kubelet[178160]: INFO 08:20:38.019896 178160 scope.go:110] "RemoveContainer" containerID="cc4b38a68d8e34264d83b0859a9b9c84b6d71b07f886ba76a9d18e6e2fa63d81"
Sep 14 08:20:38 jupyterhub-test kubelet[178160]: INFO 08:20:38.019999 178160 scope.go:110] "RemoveContainer" containerID="8ceae598793f115e91fdd87e77b087d5ce5ccede2c719ae78c3c7a3c477bf714"
Sep 14 08:20:38 jupyterhub-test kubelet[178160]: ERROR 08:20:38.088723 178160 kubelet.go:2419] "Error getting node" err="node \"jupyterhub-test\" not found"
Sep 14 08:20:38 jupyterhub-test kubelet[178160]: ERROR 08:20:38.188858 178160 kubelet.go:2419] "Error getting node" err="node \"jupyterhub-test\" not found"
