2

I stumbled upon this strange limitation when deploying more than 3 applications utilizing AWS ALB within an ingress controller.

Up to 3 Applications, there's no problem, when a 4th ingress is provisioned, it doesn't get an endpoint:

4th ingress doesn't get an endpoint

However the LoadBalancer (ALB) is provisioned as expected:

Provisioned LoadBalancers

Is there a limitation of 3 Ingresses I'm not aware of? Or did I set this limit somewhere in the configuration?

Output of kubectl describe ingress of "failing" ingress (note the missing address - the other 3 ingresses all have addreses):

Name:             some-ingress
Namespace:        default
Address:          
Default backend:  default-http-backend:80 (<none>)
Rules:
  Host  Path  Backends
  ----  ----  --------
  *     
        /    ssl-redirect-default:use-annotation (<none>)
        /*   ssl-redirect:use-annotation (<none>)
        /*   some-service:80 (192.168.92.252:8080)
Annotations:
  alb.ingress.kubernetes.io/actions.ssl-redirect:          {"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Port": "443", "StatusCode": "HTTP_301"}}
  alb.ingress.kubernetes.io/actions.ssl-redirect-default:  {"Type": "redirect", "RedirectConfig": { "Protocol": "HTTPS", "Path": "/someapp/#{path}", "Port": "443", "StatusCode": "HTTP_301"}}
  alb.ingress.kubernetes.io/certificate-arn:               arn:aws:acm:eu-central-1:...
  alb.ingress.kubernetes.io/listen-ports:                  [{"HTTP": 80}, {"HTTPS":443}]
  alb.ingress.kubernetes.io/scheme:                        internet-facing
  kubernetes.io/ingress.class:                             alb
Events:
  Type    Reason  Age                From                    Message
  ----    ------  ----               ----                    -------
  Normal  CREATE  16m                alb-ingress-controller  LoadBalancer some-alb created, ARN: some-alb-arn:loadbalancer/app/some-alb/some-ids
  Normal  CREATE  16m (x2 over 16m)  alb-ingress-controller  rule 1 created with conditions [{    Field: "path-pattern",    Values: ["/"]  }]
  Normal  CREATE  16m (x2 over 16m)  alb-ingress-controller  rule 2 created with conditions [{    Field: "path-pattern",    Values: ["/*"]  }]
  Normal  CREATE  16m                alb-ingress-controller  rule 3 created with conditions [{    Field: "path-pattern",    Values: ["/*"]  }]
Simon Z.
  • 497
  • 4
  • 11

1 Answers1

3

Found the problem, apparently there is a Security Group Limit of 5 per Cluster Node ENI (Elastic Network Interface), and every new application instance with ALB Ingress adds a SG. This is a service quota, which can be increased:

Service Quota

The relevant logs were found via:

kubectl logs -n kube-system   deployment.apps/alb-ingress-controller

I already had 2 SGs existent for the ENIs, so after 3 application instances were deployed, the limit was reached. I just increased the limit to 15 (per service quota request) and voilà, I can deploy more application instances with ALB Ingresses.

For me, this solved the issue, but this solution will not scale endlessly, as with every instance of an application a new Security Group is added to the ENIs, so 13 application instances is the new limit for me.

Simon Z.
  • 497
  • 4
  • 11
  • Thanks for the reply, this makes sense..I had worked around by nginx ingress :) .. – shivMagus Feb 29 '20 at 15:52
  • The way ALB Ingress handles security groups has changed, so this should no longer be an issue - see that issue and try release 1.1.3 or later ([changelog](https://github.com/kubernetes-sigs/aws-load-balancer-controller/releases/tag/v1.1.3) has details). AWS ALB Ingress Controller has also been renamed to AWS Load Balancer Controller (mentioned to help people find this). – RichVel Mar 20 '21 at 17:21
  • I found it took several days to get the limit increased via AWS Support, so the fix in controller is preferable. – RichVel Mar 20 '21 at 17:23