Kuberenetes Node to Node Communication Not working as expected

Question

Hello to All

I do Have problem on My Kubernete Cluster

Specification

Cluster

NAME                 STATUS   ROLES    AGE   VERSION   INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION     CONTAINER-RUNTIME
k8s-w02-prod   Ready    <none>   40d   v1.19.2   192.168.25.20   <none>        Ubuntu 20.04.1 LTS   5.4.0-54-generic   docker://19.3.8
k8s-m01-prod   Ready    master   40d   v1.19.2   10.60.17.15    <none>        Ubuntu 20.04.1 LTS   5.4.0-58-generic   docker://19.3.8
k8s-m02-prod   Ready    master   40d   v1.19.2   10.60.17.16    <none>        Ubuntu 20.04.1 LTS   5.4.0-54-generic   docker://19.3.8
k8s-m03-prod   Ready    master   40d   v1.19.2   10.60.17.17    <none>        Ubuntu 20.04.1 LTS   5.4.0-54-generic   docker://19.3.8
k8s-w01-prod   Ready    <none>   40d   v1.19.2   192.168.29.20    <none>        Ubuntu 20.04.1 LTS   5.4.0-54-generic   docker://19.3.8

Cluster Network Plugin Using Calico

Pod : networking: podSubnet: 10.65.0.0/16

Calico process is running.

IPv4 BGP status
+---------------+-------------------+-------+------------+-------------+
| PEER ADDRESS  |     PEER TYPE     | STATE |   SINCE    |    INFO     |
+---------------+-------------------+-------+------------+-------------+
| 192.168.25.20 | node-to-node mesh | up    | 23:37:55   | Established |
| 10.60.11.156  | node-to-node mesh | up    | 2021-01-04 | Established |
| 10.60.11.157  | node-to-node mesh | up    | 2021-01-04 | Established |
| 192.168.29.20 | node-to-node mesh | up    | 2021-01-04 | Established |
+---------------+-------------------+-------+------------+-------------+

IPv6 BGP status
No IPv6 peers found.

It Uses Node to Node Mesh

Problem

When i run the Simple application for example: ArgoCD

argo-cd-argocd-application-controller-74dd8b79f5-vldhb   1/1     Running   0          14h   10.65.102.48   k8s-w02-prod   <none>           <none>
argo-cd-argocd-dex-server-5c656d6c6c-shb69               1/1     Running   0          14h   10.65.102.52   k8s-w02-prod   <none>           <none>
argo-cd-argocd-redis-9757589c5-6w2p6                     1/1     Running   0          14h   10.65.102.60   k8s-w02-prod   <none>           <none>
argo-cd-argocd-repo-server-774c6856f9-vgmq8              1/1     Running   0          14h   10.65.102.4    k8s-w02-prod   <none>           <none>
argo-cd-argocd-server-669fc6db5c-x5w4k                   1/1     Running   0          13h   10.65.72.159   k8s-w01-prod   <none>           <none>

Q) I can not Access the ArgoCD Web UI Bcoz as i see Pods are running on

Worker01:k8s-w01-prod 192.168.25.20

ip route | grep tun

10.65.69.192/26 via 10.60.17.17 dev tunl0 proto bird onlink 
10.65.102.0/26 via 192.168.25.20 dev tunl0 proto bird onlink 
10.65.187.64/26 via 10.60.17.15 dev tunl0 proto bird onlink 
10.65.233.192/26 via 10.60.17.16 dev tunl0 proto bird onlink

worker02:k8s-w02-prod 192.168.29.20

10.65.69.192/26 via 10.60.17.17 dev tunl0 proto bird onlink 
10.65.72.128/26 via 192.168.29.20 dev tunl0 proto bird onlink 
10.65.187.64/26 via 10.60.17.15 dev tunl0 proto bird onlink 
10.65.233.192/26 via 10.60.17.16 dev tunl0 proto bird onlink

Runs on Different Subnet

Ping Works Completely OK on both Side

When i Use the Node Selector labels to run pods on selected Node i.e worker01 or worker02 then the issue is solved.

Q) How can i route the Traffic so that the Application ArgoCD WebUI works without any node-node communication problem(Pod can run on any Node and it can communicate each other)

Q) Is it good Idea to use BGP on calico? with out Node-to-Node mesh

https://docs.projectcalico.org/networking/bgp

Please Advice me how can i fix this issue

Hello, I have a difficulty to understand your question. Could you clarify that you can communicate `Pods` between themselves when they are placed on different nodes (`k8s-w01-prod` -> `k8s-w02-prod`). The [documentation](https://kubernetes.io/docs/concepts/cluster-administration/networking/#the-kubernetes-network-model) states that: `pods on a node can communicate with all pods on all nodes without NAT` and it seems like you have NAT by the internal IP addresses. — Dawid Kruk, Jan 11 '21 at 13:38
the Question is I can not Communicate the Pods via different Worker. Which Mean if i run argocd application i need 5 pods to be up and run. if the pods get distributed among workers then i can not get proper communication for example: argo-cd-argocd-dex-server-5c656d6c6c-shb69 is ldap synn for argocd server which argo-cd-argocd-server-669fc6db5c-x5w4k when i add a ldap to argocd the argocd-server get communicated with DEX-server. this communication is not working its keep on loading ..... — Rehan, Jan 11 '21 at 17:55
As per your last comment, it looks like your Kubernetes cluster have connectivity issues. I'm worried about the differences in the IP addresses of your nodes. Each of your nodes is in the different network? Are the workers in the same network? — Dawid Kruk, Jan 15 '21 at 09:16
yes all the works are on different network bcoz we switch works on different data center location. and about Calico https://docs.projectcalico.org/v3.14/manifests/calico.yaml — Rehan, Jan 15 '21 at 15:40
I tested with https://raw.githubusercontent.com/cilium/cilium/1.7.0/install/kubernetes/quick-install.yaml it is much faster andso for all looks good — Rehan, Jan 15 '21 at 15:41

score 0 · Answer 1 · answered Jan 09 '21 at 23:48

I found Work around to Work with this problem by using nodeAffinity

       affinity:   
         nodeAffinity:
           preferredDuringSchedulingIgnoredDuringExecution:
           - weight: 1
             preference:
               matchExpressions:
               - key: "kubernetes.io/hostname"
                 operator: In
                 values:
                 - k8s-w01-prod

But Still the Problem exist on my side

Kuberenetes Node to Node Communication Not working as expected

1 Answers1