1

I have an application running on Kubernetes, that uses nginx as the ingress controller that created a load balancer in AWS. I noticed that by default the application is open to the World, with 0.0.0.0/32 is added to the inbound rules of the AWS Security group that is attached to the load balancer. I want to allow only certain IPs to access the application. That makes me use nginx.ingress.kubernetes.io/whitelist-source-range annotation in the ingress controller.

But I wouldn't know the IPs of the entities that must be allowed to access the application beforehand. An upstream process (Jenkins job) that creates certain containers, which try to talk to the application that's running on Kube.

How can I dynamically modify the ingress controller annotation to add and remove IPs without causing any downtime? And No, I do not have a common IP range that I can add. I have several different VPCs which have their own CIDR blocks.

M S Puranik
  • 35
  • 7

1 Answers1

0

Short answer: you don't put the whitelist annotation on the controller, you put it on the ingress resource. And updating that does not require any downtime.

--

Long answer: Yes, by default, the controller loadbalancer is open to the world, and that is expected. All traffic comes into the ingress controller's loadbalancer and the controller then determines how to route it within the cluster.

It determines this routing via the use of ingress resources. Here is an example:


apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: nginx
    nginx.ingress.kubernetes.io/whitelist-source-range: 12.0.0.0/8,10.0.0.0/8
  name: ingress-whitelist-example
spec:
  rules:
  - host: somehost.com
    http:
      paths:
      - backend:
          service:
            name: service-for-somehost
            port:
              number: 80
  tls:
  - hosts:
    - somehost.com
    secretName: tls-secret

The ingress controller will get traffic for (in this example) the domain host somehost.com and it will route the request to the service service-for-somehost in the cluster, on port 80

If a request comes into the controller outside of the ranges 12.0.0.0/8 or 10.0.0.0/8 (as described by the annotation), then the controller will reject the request with a Forbidden error.

The ingress resource is not a resource that needs to be respun or taken down for updates, like Deployments.

The AWS load balancer is external to the cluster. You can, of course, choose to block/whitelist traffic within the AWS cloud before it reaches the cluster and that is fine, but that is not managed within the nginx controller.

Some further reading in Ingresses is available here

Blender Fox
  • 4,442
  • 2
  • 17
  • 30
  • Ingress resource in my case is AWS load balancer? – M S Puranik Feb 01 '22 at 11:57
  • I have expanded the answer to cover your follow-up question – Blender Fox Feb 01 '22 at 12:31
  • Thank you. But for some reason, even though I add/edit IP ranges in the Security group's inbound rules, it is getting overridden by the Kubernetes scheduler job, which replaces all rules with 0.0.0.0/32. I think kube is detecting the change in security group as a deviation from the desired state and trying to re-apply the actual state. – M S Puranik Feb 01 '22 at 13:26
  • 1
    That is possible, you may want to look into the available AWS ELB annotations you set via the controller service: https://kubernetes.io/docs/concepts/services-networking/service/. I think the `service.beta.kubernetes.io/aws-load-balancer-extra-security-groups` would work for you -- you can attach security groups to the ELB that will block anything not allowed. – Blender Fox Feb 01 '22 at 13:30