1

We have a OpenShift 4.8 cluster with 3 master nodes and 10 worker nodes in Azure. All the worker and master nodes are added under the same load balancer. I am a bit confused about how ingress traffic reaches the cluster. When someone accesses the DNS of their application, traffic comes through the load balancer over port 80/443 to any of the cluster nodes(including the master). But the ingress controller pods or running only on one or two nodes. How exactly traffic reaches to the correct ingress controller pods? Also once the traffic reaches the node how exactly it identifies the correct ingress host to forward traffic to? Another question around this is, why both master and worker nodes are added under the same load balancer?

Hound
  • 837
  • 17
  • 31
  • I think this doc has what you're looking for: https://kubernetes.io/docs/concepts/services-networking/service/#proxy-mode-iptables. Per your last question, that sounds like a misconfig as master nodes should not be added to an ingress load balancer. – jordanm Oct 08 '22 at 16:03

2 Answers2

1

How exactly traffic reaches to the correct ingress controller pods? Also once the traffic reaches the node how exactly it identifies the correct ingress host to forward traffic to?

The ingress controller doesn't need to deploy on every compute node because it knows all the way to your pods which has a route.

How to know which nodes are avaiable

Load Balancer has a health check feature to check port or http request on a node. That helps to know available nodes the ingress pods work on.

How to reach the ingress controller

The ingress opens ports in the pod, not the node. OpenShift in a cloud provider like Azure deploys load balancer service for the ingress. That deploys Load balancer in Azure and binds ports on the node(host) to receive requests from outside OpenShift cluster. Those ports are defined randomly. The load balancer service makes setting up the load balancer in Azure to reach the ports on the nodes. So, you don't need to worry about which ports on nodes are opened.

How to transfer requests to the correct pods

The ingress controller consists of HAProxy which works as L7 proxy mode. A request to the ingress controller should have 'host name' and it should be matched a route you defined. That allows to lead your request to your correct pod.

Another question around this is, why both master and worker nodes are added under the same load balancer?

The ingress controller is a pod so if you don't specify 'Node Selector', the pod can be deployed any nodes in an OpenShift Cluster. Since the pods could be deployed different node accidentally, Load Balancer is prepared for it.

hiroyukik
  • 713
  • 1
  • 6
  • 14
  • I agree on the L7 Proxy mode and SNI based routing to ingress. But the confusion I have is how exactly the traffic from LoadBalancer gets routed to the exact nodes that run ingress controller pods?. Once the ingress controller received traffic, everything you explained i agree with. – Hound Oct 09 '22 at 03:08
  • Load Balancer has a health check feature to check port or http request on a node. That helps to know available nodes the ingress pods work on. – hiroyukik Oct 09 '22 at 05:19
  • Yeah there is a health check configured for /healthz over port 30331 and I can see the in service yaml has an entry of ```healthCheckNodePort: 30331``` but that is different from the nodePort allocated for 80 or 443. I tried calling /healthz endpoint within pod over 30331 but did not work. I guess I am a bit confused still about how this health check is working – Hound Oct 09 '22 at 18:19
  • I have now checked, and from all nodes curl :30331/healthz check succeeds and returns 200 OK. Not just the ones having ingress controller pods. Still wondering how the traffic exactly goes to the node that has the ingress controller pod running – Hound Oct 09 '22 at 20:31
  • The ingress opens ports in the pod, not the node. Openshift in a cloud provider like Azure deploys load balancer service for the ingress. That deploys Load balancer in Azure and binds ports on the node(host) to receive requests from outside OpenShift cluster. Those ports are defined randomly. The load balancer service makes setting up the load balancer in Azure to reach the ports on the nodes. So, you don't need to worry about which ports on nodes are opened. – hiroyukik Oct 09 '22 at 23:32
  • "That deploys Load balancer in Azure and binds ports on the node(host) to receive requests from outside OpenShift cluster." which I completely agree with, but what I still did not understand or you did not answer is, how the exact node that runs the ingress controller pod receives the traffic and all other nodes reject it. It is definitely not the load balancer health check, if the health check fails node will be marked as unstable. I hope I made the question clear – Hound Oct 11 '22 at 01:01
  • Load Balancer only transfers requests to the nodes which is healthy. And if you send a request to a node which the ingress controller doesn't running on, there are no process to listen ports for HTTP, HTTPS and healthcheck. So no one could receive the requests then, refused the connections. Is that helpful? – hiroyukik Oct 11 '22 at 02:12
  • 'Load Balancer only transfers requests to the nodes which is healthy". True, this is determined using the /healthz endpoint check performed by the load balancer and if the health check fails, the nodes will be marked as unhealthy. In my case all 10 nodes are healthy and all ten nodes are actively listening on 30331 as well. – Hound Oct 11 '22 at 16:59
  • In my environment, I got 503 error when I called `curl http://:30331/healthz` to a node which Ingress is not running. Whereas I got 200 status code when I called the command to a node which ingress is running. I'm not sure how to check the nodes are healthy as a backend on Azure Portal. However, Azure Load Balancer could know which node is available. – hiroyukik Oct 14 '22 at 04:32
  • Azure checks nodes are healthy based on the health endpoint you configured. As long as the endpoint returns 200 ok, Loadbalancer will mark the node as health. On the node that gave you 503, can you do a ```netstat -nalp | grep 30331``` (assuming your health check port also is 30331) – Hound Oct 14 '22 at 20:26
0

If anyone lands here looking for the answer, there is a iptable rule on the node that forwards the packets to the service

Ingress service has the nodePort 30331. Grep the ip table using the port number

# iptables -t nat -L KUBE-NODEPORTS -n  | column -t | grep "30331"
KUBE-MARK-MASQ             tcp             --   127.0.0.0/8  0.0.0.0/0    /*  ingress/default:https         */  tcp  dpt:30331
KUBE-XLB-MBAZS3WDHL45BPIZ  tcp             --   0.0.0.0/0    0.0.0.0/0    /*  ingress/default:https         */  tcp  dpt:30331

My service ip address is: 172.70.92.82. Grep the ip table using service ip

#  iptables -t nat -L KUBE-SERVICES -n  | column -t | grep "172.70.92.82"
KUBE-SVC-HEVFQXAKPPGAL4BV  tcp            --   0.0.0.0/0    172.70.92.82   /*  ingress/default:http    cluster       IP          */     tcp   dpt:80                                                                          
KUBE-SVC-MBAZS3WDHL45BPIZ  tcp            --   0.0.0.0/0    172.70.92.82   /*  ingress/default:https    cluster      IP          */     tcp   dpt:443
Hound
  • 837
  • 17
  • 31