kubernetes v1.18.8 installation issue

Question

I have deployed Kubernetes cluster v1.18.8 with kubeadm on production environment.Cluster setup is 3 Master and 3 Worker nodes with external Kube-api loadbalancer, etcd residing in Master nodes.Didn't see any issue during installation and all pods in kube-system are running. However when i get error when i run below command i get error:

  kubectl get cs
    NAME                 STATUS      MESSAGE                                                                                     ERROR
    controller-manager   Unhealthy   Get http://127.0.0.1:10252/healthz: dial tcp 127.0.0.1:10252: connect: connection refused
    scheduler            Unhealthy   Get http://127.0.0.1:10251/healthz: dial tcp 127.0.0.1:10251: connect: connection refused
    etcd-0               Healthy     {"health":"true"}

While troubleshooting i found that the ports are not being listened.

    sudo netstat -tlpn |grep kube
tcp        0      0 127.0.0.1:10248         0.0.0.0:*               LISTEN      132584/kubelet
tcp        0      0 127.0.0.1:10249         0.0.0.0:*               LISTEN      133300/kube-proxy
tcp        0      0 127.0.0.1:10257         0.0.0.0:*               LISTEN      197705/kube-control
tcp        0      0 127.0.0.1:10259         0.0.0.0:*               LISTEN      213741/kube-schedul
tcp6       0      0 :::10250                :::*                    LISTEN      132584/kubelet
tcp6       0      0 :::6443                 :::*                    LISTEN      132941/kube-apiserv
tcp6       0      0 :::10256                :::*                    LISTEN      133300/kube-proxy

If i check the same thing on development enviroment kubernetes cluster(v1.17) i see no issue.

kubectl get cs
NAME                 STATUS    MESSAGE             ERROR
controller-manager   Healthy   ok
scheduler            Healthy   ok
etcd-0               Healthy   {"health":"true"}

sudo netstat -tlpn |grep 102
tcp        0      0 127.0.0.1:10257         0.0.0.0:*               LISTEN      2141/kube-controlle
tcp        0      0 127.0.0.1:10259         0.0.0.0:*               LISTEN      2209/kube-scheduler
tcp        0      0 127.0.0.1:10248         0.0.0.0:*               LISTEN      1230/kubelet
tcp        0      0 127.0.0.1:10249         0.0.0.0:*               LISTEN      2668/kube-proxy
tcp6       0      0 :::10256                :::*                    LISTEN      2668/kube-proxy
tcp6       0      0 :::10250                :::*                    LISTEN      1230/kubelet
tcp6       0      0 :::10251                :::*                    LISTEN      2209/kube-scheduler
tcp6       0      0 :::10252                :::*                    LISTEN      2141/kube-controlle

On newly created prodction cluster i have deployed nginx and another application just to test how the kubernetes components behave, didn't see any error. Is it the expected behaviour in version v1.18? Will really apprecite any help on this.

NOTE: No port is being blocked in internal communication

What are you using as client to create the kubernetes cluster? — Fahima Mokhtari, Sep 05 '20 at 19:37
I am using Kubeadm(v1.18.8) to deploy the cluster on On-premises VMs. — Mr Kashyap, Sep 06 '20 at 06:32
Maybe the problem lies in the Kubeadm version. Try to use another version or use a different client such as kops, in my case, kops woked perfectly for me. — Fahima Mokhtari, Sep 06 '20 at 08:28
If there's a change then it should be mentioned in the kubernetes document, What version you are working on? — Mr Kashyap, Sep 06 '20 at 09:23
In my case,I was using ubuntu, kops version 1.17, kubernetes v1.17.8. — Fahima Mokhtari, Sep 06 '20 at 09:46
I don't have issue with v1.17.x either. This must be something with version 1.18.8 — Mr Kashyap, Sep 06 '20 at 09:58
@MrKashyap Could you please provide the logs from kube-scheduler and kube-controller-manager? — Wytrzymały Wiktor, Sep 07 '20 at 09:08

score 1 · Answer 1 · answered Sep 09 '20 at 17:52

The command Kubectl get componentstatus is depreciated in newer version(1.19) and it already has many issues.

The main point to note here is that Kubernetes has disabled insecure serving of these components for older versions(atleast from v1.18). Hence i couldn't see kube-controller and kube-scheduler being listned on 1051 and 1052 ports. To restore this functionality you can remove the --port=0 from their manifests files(Not recommended as this can expose their metrics to the whole internet) that you can see inside:

/etc/kubernetes/manifests/

I commented out --port=0 field from the manifest file just to check this, kubectl get componentstatus command worked.

kubernetes v1.18.8 installation issue

1 Answers1