11

I have set up the nginx ingress controller following this guide.

The ingress works well and I am able to visit the defaultbackend service and my own service as well.

But when reviewing the objects created in the Google Cloud Console, in particular the load balancer object which was created automatically, I noticed that the health check for the other nodes are failing: enter image description here

Is this because the ingress controller process is only running on the one node, and so it's the only one that passes the health check? How do I make the other nodes pass?

Jesse Scherer
  • 1,492
  • 9
  • 26
john2x
  • 22,546
  • 16
  • 57
  • 95

1 Answers1

13

Your assumption is correct. The healthy node is indeed the one running the nginx pod.

The guide you're using configures the service with externalTrafficPolicy: Local.(https://raw.githubusercontent.com/kubernetes/ingress-nginx/master/deploy/provider/gce-gke/service.yaml)

That policy configures kube-proxy to never route traffic for a service off of the node where it arrives. So, when the load balancer traffic reached the nodes that have no nginx pod, the health check failed and the load balancer stopped sending traffic to them.

This configuration had the advantage of avoiding an extra network hop to get to the nginx pod. If you need more nodes to handle the traffic, you can ensure that there are nginx pods running there too. If you don't mind the extra network hop, you can change the externalTrafficPolicy too.

coreypobrien
  • 1,921
  • 17
  • 17
  • Hmm I'm not sure I follow how setting `externalTrafficPolicy` to `Cluster` would make the other nodes healthy. So when the health check runs on a node without the nginx pod, it causes the request to make hops until it reaches the node with the nginx pod, thus making the original node healthy? – john2x Jan 12 '18 at 13:19
  • That's correct. The normal behavior for external traffic is that every node opens the same port. The traffic then follows iptables rules set up by kube-proxy to reach one specific pod on any node that matches the selector for the service. – coreypobrien Jan 12 '18 at 17:22
  • Does this also mean that load balancing isn't actually happening with `externalTrafficPolicy: Local` if only 1 node (the one where the nginx ingress controller is running) can accept incoming connections? This feels like a bit of a showstopper if you want to whitelist IPs (preserving original source IP with `Local`) but also have the benefits of the LB. – pappy-o Mar 27 '18 at 21:36
  • That is correct. The last paragraph of my answer addresses this. You get the benefit of skipping extra network hops and still get load balancing by running nginx replicas on the other nodes in the cluster. – coreypobrien Mar 28 '18 at 20:10
  • Got it; thanks! Following your other nugget of advice [here](https://stackoverflow.com/a/47961015/6051841), I scaled up my ingress controller to match the number of nodes, and all healthchecks in the LB are green. I was concerned that all replicas would end up on the same node, but it's working beautifully so far. Thanks so much, @coreypobrien – pappy-o Mar 28 '18 at 23:44