Ingress-nginx controller does not run ELB properly in AWS EC2 Kubernetes Cluster

Question

I create cluster with kops utility on AWS EC2. Right now I am trying to configure ingress-nginx controller so it routes all traffic in my cluster. I need it handles HTTP, HTTPS and WebSocket connections. Based on this guide I made all required things:

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/mandatory.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/provider/aws/service-l4.yaml
kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/nginx-0.28.0/deploy/static/provider/aws/patch-configmap-l4.yaml

When I am trying to get all items in ingress-nginx namespace with kubectl -n ingress-nginx get all:

NAME                    TYPE           CLUSTER-IP    EXTERNAL-IP                                                                 PORT(S)                      AGE
service/ingress-nginx   LoadBalancer   100.71.94.9   a7d3fe1383e344c1d8cb2de671xxxxxx-810xxxxxx.eu-central-1.elb.amazonaws.com   80:32389/TCP,443:31803/TCP   16m

When I open AWS console -> EC2 Instances -> Load Balancer, I can see that ELB has been created, but there OutOfService status on each node under "Instances" tab. So I can't get reach my ELB URL: a7d3fe1383e344c1d8cb2de671xxxxxx-810xxxxxx.eu-central-1.elb.amazonaws.com:

There is some more details about service using kubectl -n ingress-nginx describe service/ingress-nginx

Name:                     ingress-nginx
Namespace:                ingress-nginx
Labels:                   app.kubernetes.io/name=ingress-nginx
                          app.kubernetes.io/part-of=ingress-nginx
Annotations:              kubectl.kubernetes.io/last-applied-configuration:
                            {"apiVersion":"v1","kind":"Service","metadata":{"annotations":{"service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout":"60"...
                          service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: 60
                          service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: *
Selector:                 app.kubernetes.io/name=ingress-nginx,app.kubernetes.io/part-of=ingress-nginx
Type:                     LoadBalancer
IP:                       100.71.94.9
LoadBalancer Ingress:     a7d3fe1383e344c1d8cb2de671xxxxxx-810xxxxxx.eu-central-1.elb.amazonaws.com
Port:                     http  80/TCP
TargetPort:               http/TCP
NodePort:                 http  32389/TCP
Endpoints:                <none>
Port:                     https  443/TCP
TargetPort:               https/TCP
NodePort:                 https  31803/TCP
Endpoints:                <none>
Session Affinity:         None
External Traffic Policy:  Cluster
Events:
  Type    Reason                Age   From                Message
  ----    ------                ----  ----                -------
  Normal  EnsuringLoadBalancer  15m   service-controller  Ensuring load balancer
  Normal  EnsuredLoadBalancer   15m   service-controller  Ensured load balancer

Am I missed something?

UPD #1

If I do the same things in EKS cluster, everything works well and ingress-controller appears on each node. Any ideas..?

Arghya Sadhu · Answer 1 · 2020-02-04T14:45:19.360

1

You need to add security group on the EC2 instances(kubernetes worker nodes) where you have the nginx deployed to allow port 80 and 443 for the security group that was created for ELB.

Edit:

The endpoints section of the service/ingress-nginx service does not have IPs of the nginx pods. Hence when ELB sends a health check request but the request can not reach the pods so health check fails and ELB marks the backend as outofservice.

edited Feb 04 '20 at 14:45

answered Feb 03 '20 at 13:11

Arghya Sadhu

41,002
9
78
107

I have this kind of security group. And it has all required settings by default. – matterai Feb 03 '20 at 13:19
Do you also have outbound rules on the security group of the load balancer? – Arghya Sadhu Feb 03 '20 at 13:25
Yes. All traffic : All : All : 0.0.0.0/0. – matterai Feb 03 '20 at 13:31
Just to confirm "All : All : 0.0.0.0/0" this is in the inbound or outbound ? – Arghya Sadhu Feb 03 '20 at 13:53
Outbound. This security group creates automatically when you run `kubectl apply -f .../service-l4.yaml`. – matterai Feb 03 '20 at 13:56
can you verify nginx pods are running? – Arghya Sadhu Feb 03 '20 at 14:00
They are not.. And I do not know why, actually :( – matterai Feb 03 '20 at 14:01
Can you describe the nginx pods and provide that output? – Arghya Sadhu Feb 03 '20 at 14:02
@ArghyaSadhu thank you, your answer helped me a lot, I was looking for that since yesterday – d3vpasha Nov 23 '22 at 09:01

Ingress-nginx controller does not run ELB properly in AWS EC2 Kubernetes Cluster

1 Answers1