2

I have got a simple pod up and running on my worker node which I am trying to ping from my master node. I'm getting 100% packet loss.

What's working?

  • The worker node itself can be pinged from the master node.
  • The pod inside the worker node can be pinged from inside of the worker node.

What's not working?

  • The master node trying to ping worker node's pod.

My take

I think there's a calico misconfiguration that has happened at the master node as the master node doesn't quite understand the IP of the pod.

Results of tracepath to the pod IP

masternode@Master:~$ tracepath 192.168.171.67
 1?: [LOCALHOST]                      pmtu 1440
 1:  no reply
 2:  no reply
 3:  no reply

Please help.

Rico
  • 58,485
  • 12
  • 111
  • 141
Aakash Verma
  • 3,705
  • 5
  • 29
  • 66
  • 1
    Can you please provide more details and minimal reproducible example? How did you bootstrapped your cluster? AKS, Kubeadm? What operating system are running on those vms? You mentioned that before it was working and that you suspect calico. Have performed any changes on its config? – acid_fuji Jul 16 '20 at 10:58

1 Answers1

2

The Kubernetes pod network is not reachable from the control/data plane nodes. You can reach any Pod including control plane Pods between each other but not anything on the host.

The quoted Kubernetes Networking fundamentals from the docs:

  • Pods on a node can communicate with all pods on all nodes without NAT
  • Agents on a node (e.g. system daemons, kubelet) can communicate with all pods on that node. -- Note you can do this with docker for example, but not using the pod network.

Also, from the docs:

  • Containers within a Pod use networking to communicate via loopback.
  • Cluster networking provides communication between different Pods.
  • The Service resource lets you expose an application running in Pods to be reachable from outside your cluster.
  • You can also use Services to publish services only for consumption inside your cluster.
Aakash Verma
  • 3,705
  • 5
  • 29
  • 66
Rico
  • 58,485
  • 12
  • 111
  • 141
  • To "The Kubernetes pod network is not reachable from the control/data plane nodes." I'm sorry but I did happen to be able to ping/curl my pods in worker nodes from master nodes before. How was that happening? – Aakash Verma Jul 16 '20 at 09:48
  • 1
    You can do it from a pod in the master node. But not from the host. – Rico Jul 16 '20 at 13:45
  • 1
    This is what I was missing! A big hole in my concept, I was thinking control plane nodes can access pods on worker nodes. Thanks a lot. – Aakash Verma Jul 16 '20 at 18:28
  • One final question, I have both my master and worker node untainted so when I try to deploy a pod standing on the master node, the kube-scheduler places it on the worker node. May I ask why? The master node has more mem and compute resources than the worker node. – Aakash Verma Jul 16 '20 at 20:15
  • I think it chooses the worker nodes first unless you configure the nodeSelector (node affinity) to go to the master. Maybe your workers have a larger percentage of resources at the time the pod gets scheduled. – Rico Jul 16 '20 at 20:39