0

I'm trying to deploy Kubernetes with Calico (IPIP) with Kubeadm. After deployment is done I'm deploying Calico using these manifests

kubectl apply -f https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/rbac-kdd.yaml
kubectl apply -f https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml

Before applying it, I'm editing CALICO_IPV4POOL_CIDR and setting it to 10.250.0.0/17 as well as using command kubeadm init --pod-cidr 10.250.0.0/17.

After few seconds CoreDNS pods (for example getting addr 10.250.2.2) starts restarting with error 10.250.2.2:8080 connection refused.

Now a bit of digging:

from any node in cluster ping 10.250.2.2 works and it reaches pod (tcpdump in pod net namespace shows it).

from different pod (on different node) curl 10.250.2.2:8080 works well

from any node to curl 10.250.2.2:8080 fails with connection refused

Because it's coredns pod it listens on 53 both udp and tcp, so I've tried netcat from nodes

nc 10.250.2.2 53 - connection refused nc -u 10.250.2.2 55 - works

Now I've tcpdump each interface on source node for port 8080 and curl to CoreDNS pod doesn't even seem to leave node... sooo iptables?

I've also tried weave, canal and flannel, all seem to have same issue.

I've ran out of ideas by now...any pointers please?

inc0
  • 215
  • 2
  • 8
  • one update `dig +notcp @10.250.0.2 www.google.com` doesn't seem to be working as well, it times out – inc0 Jan 29 '19 at 20:52

1 Answers1

1

Seems to be a problem with Calico implementation, CoreDNS Pods are sensitive on the CNI network Pods successful functioning. For proper CNI network plugin implementation you have to include --pod-network-cidr flag to kubeadm init command and afterwards apply the same value to CALICO_IPV4POOL_CIDR parameter inside calico.yml.

Moreover, for a successful Pod network installation you have to apply some RBAC rules in order to make sufficient permissions in compliance with general cluster security restrictions, as described in official Kubernetes documentation:

For Calico to work correctly, you need to pass --pod-network-cidr=192.168.0.0/16 to kubeadm init or update the calico.yml file to match your Pod network. Note that Calico works on amd64 only.

kubectl apply -f https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/rbac-kdd.yaml
kubectl apply -f https://docs.projectcalico.org/v3.3/getting-started/kubernetes/installation/hosted/kubernetes-datastore/calico-networking/1.7/calico.yaml

In your case I would switched to the latest Calico versions at least from v3.3 as given in the example.

If you've noticed that you run Pod network plugin installation properly, please take a chance and update the question with your current environment setup and Kubernetes components versions with a health statuses.

Nick_Kh
  • 5,089
  • 2
  • 10
  • 16
  • Thanks, I've changed versions but it seems problem persists. I took a look at old manifest and it actually uses calico node master image, but I've tied it to 3.3 now. – inc0 Jan 30 '19 at 14:38
  • yes, nothing really informative. There were some errors about IPIP device not present but were fixed on retry and basically after few seconds Calico seems to be happy. – inc0 Jan 31 '19 at 18:39
  • Do you still observe Calico Pod failure? – Nick_Kh Feb 04 '19 at 09:26
  • Calico pod never really failed. It's all working and pod-to-pod traffic is fine. Liveness probes and readiness probes are only things that seems affected – inc0 Feb 05 '19 at 21:18
  • Do you have port 8080 to be exposed on the Pod's node host? – Nick_Kh Feb 07 '19 at 09:53
  • Node host - no, inside pod - yes, pod-to-pod works well. I've tried to `nc -l 8080` on host but it doesn't seem to be catching these requests either – inc0 Feb 08 '19 at 06:22
  • I suppose that port 8080 is not exposed to be accessed from the relevant Node, hope that you can afford it by using `NodePort` service type in a particular Pod service object, find more info [here](https://kubernetes.io/docs/concepts/services-networking/service/#nodeport). – Nick_Kh Feb 13 '19 at 09:27
  • that's not the case unfortunately. Pod to pod traffic works and I've made sure it's exposed in deployment too (even tho I think port doesn't have to be exposed in container for liveness probes). Thanks for reply, I believe I just have some weirdness coming from kubeadm – inc0 Feb 14 '19 at 23:39