Kubernetes/CoresOS Nodes cannot communicate to ApiServer Kube Proxy issues

Question

I followed this guide https://coreos.com/kubernetes/docs/latest/deploy-workers.html to setup a Kubernetes network and when I ran the Dashboard or DNS add-ons they resulted in crashloopbackoff after 25 seconds. I then followed this https://github.com/kubernetes/dashboard/blob/master/docs/user-guide/troubleshooting.md which timed out at when I executed

kubectl exec test-701078429-s5kca -- curl -k https://10.3.0.1

note if I executed kubectl exec test-701078429-s5kca -- curl -k https://192.168.3.240 it worked (public IP)

also if I executed curl -k https://10.3.0.1 on the master node is also worked.

So the problem is that my Kubelets on 10.2.0.0/16 can not communicate with the api server on 10.3.0.1/24 which suggests it may be a kube proxy issue. FYI: I can sucessfully ping from inside one node on one machine to another node on a different machine using 10.2.x.x

score 0 · Answer 1 · answered May 17 '17 at 12:26

The problem was a configuration issue in kubelets /etc/kubernetes/manifests/kube-proxy.yaml

the line - --master=https:// requires https:// wher I just has the IP 192.168.3.220. Below is my working kube-proxy.yaml

    apiVersion: v1
    kind: Pod
    metadata:
      name: kube-proxy
      namespace: kube-system
    spec:
      hostNetwork: true
      containers:
      - name: kube-proxy
        image: quay.io/coreos/hyperkube:v1.6.1_coreos.0
        command:
        - /hyperkube
        - proxy
        - --master=https://192.168.3.240
        - --kubeconfig=/etc/kubernetes/worker-kubeconfig.yaml
        - --proxy-mode=iptables
        securityContext:
          privileged: true
        volumeMounts:
        - mountPath: /etc/ssl/certs
          name: "ssl-certs"
        - mountPath: /etc/kubernetes/worker-kubeconfig.yaml
          name: "kubeconfig"
          readOnly: true
        - mountPath: /etc/kubernetes/ssl
          name: "etc-kube-ssl"
          readOnly: true
      volumes:
      - name: "ssl-certs"
        hostPath:
          path: "/usr/share/ca-certificates"
      - name: "kubeconfig"
        hostPath:
          path: "/etc/kubernetes/worker-kubeconfig.yaml"
      - name: "etc-kube-ssl"
        hostPath:
          path: "/etc/kubernetes/ssl"

The guide which is great, was just a little misleading in this area as in the immediate code snippet prior to it for /etc/systemd/system/kubelet.service it had --api-servers=https://${MASTER_HOST} \ with the https:// present where as the yaml for /etc/kubernetes/manifests/kube-proxy.yaml has - --master=${MASTER_HOST} without https:// prefix

Kubernetes/CoresOS Nodes cannot communicate to ApiServer Kube Proxy issues

1 Answers1