1

I have 3 ingress namespace in AKS Cluster. One of standart ingress deploy file, the other two are internal and external modified versions of the original file. So right now I have 3 ingress and 3 external ip(one of internal with adding service annotations this line service.beta.kubernetes.io/azure-load-balancer-internal: "true")

    $ kubectl get svc -A
external-ingress   ingress-nginx-controller             LoadBalancer   10.245.57.76     3.3.3.3   80:32112/TCP,443:31761/TCP   3h24m
external-ingress   ingress-nginx-controller-admission   ClusterIP      10.245.28.35     <none>    443/TCP                      3h24m
ingress-nginx      ingress-nginx-controller             LoadBalancer   10.245.12.12     1.1.1.1   80:31716/TCP,443:32023/TCP   40m
ingress-nginx      ingress-nginx-controller-admission   ClusterIP      10.245.110.233   <none>    443/TCP                      40m
internal-ingress   ingress-nginx-controller             LoadBalancer   10.245.173.35    2.2.2.2   80:32567/TCP,443:30296/TCP   3h25m
internal-ingress   ingress-nginx-controller-admission   ClusterIP      10.245.39.250    <none>    443/TCP                      3h25m

I want to use two of ingress but it doesn't work as i think. I try to manage with " ingressClassName: " but it's working unexpended.

$ kubectl get ing -w
NAME               CLASS              HOSTS         ADDRESS   PORTS     AGE
external-ingress   nginx              test.io       1.1.1.1   80, 443   3h4m
internal-ingress   internal-ingress   admin.test.io 1.1.1.1   80        3h4m
external-ingress   nginx              test.io       2.2.2.2   80, 443   3h5m
external-ingress   nginx              test.io       3.3.3.3   80, 443   3h5m
external-ingress   nginx              test.io       1.1.1.1   80, 443   3h5m
external-ingress   nginx              test.io       2.2.2.2   80, 443   3h6m
external-ingress   nginx              test.io       3.3.3.3   80, 443   3h6m
external-ingress   nginx              test.io       1.1.1.1   80, 443   3h6m
external-ingress   nginx              test.io       2.2.2.2   80, 443   3h7m
external-ingress   nginx              test.io       3.3.3.3   80, 443   3h7m
external-ingress   nginx              test.io       1.1.1.1   80, 443   3h7m
external-ingress   nginx              test.io       2.2.2.2   80, 443   3h8m
external-ingress   nginx              test.io       3.3.3.3   80, 443   3h8m

When I changed the "ingressClassName: nginx" External IP constantly changing. When I changed "ingressClassName: external-ingress" IP sometimes changing, sometimes not changing.. and when it's chaning it's not work..

original deployment file

kubectl apply -f https://raw.githubusercontent.com/kubernetes/ingress-nginx/controller-v1.3.0/deploy/static/provider/cloud/deploy.yaml

I create a external-ingress and internal-ingress with this file changing "ingress-nginx" line to "internal-nginx" and "external-nginx".

Why is the ip constantly changing?

thargalin
  • 123
  • 1
  • 2
  • 16
  • When you say `When I changed the "ingressClassName: nginx" External IP constantly changing. When I changed "ingressClassName: external-ingress" IP sometimes changing, sometimes not changing.. and when it's chaning it's not work..`, can you provide more information about what you exactly see? This might be due to duplication of hostnames. How are IPs being assigned to your ingresses? – zer0 Jul 16 '22 at 05:23

3 Answers3

1

If the ingress resource is regularly changing external IPs, this indicates you have more than one ingress controller in the cluster and have not specified an ingress class on the ingress resource.

As such, all ingress controllers in the cluster (which have not been restricted to namespace) will try to "fight" for management of the traffic going through that ingress, each time the winning controller will set the IP of the ingress to their own loadbalancer IP.

This is why you're seeing the IP of your ingress change to 1.1.1.1, 2.2.2.2, and 3.3.3.3 because these are the ingress controllers you have installed in your cluster, as per your snippet:

$ kubectl get svc -A
external-ingress   ingress-nginx-controller             LoadBalancer   10.245.57.76     3.3.3.3   80:32112/TCP,443:31761/TCP   3h24m
external-ingress   ingress-nginx-controller-admission   ClusterIP      10.245.28.35     <none>    443/TCP                      3h24m
ingress-nginx      ingress-nginx-controller             LoadBalancer   10.245.12.12     1.1.1.1   80:31716/TCP,443:32023/TCP   40m
ingress-nginx      ingress-nginx-controller-admission   ClusterIP      10.245.110.233   <none>    443/TCP                      40m
internal-ingress   ingress-nginx-controller             LoadBalancer   10.245.173.35    2.2.2.2   80:32567/TCP,443:30296/TCP   3h25m
internal-ingress   ingress-nginx-controller-admission   ClusterIP      10.245.39.250    <none>    443/TCP                      3h25m

You can fix this by ensuring all your ingress controllers are set to only watch for a specific ingress class and/or restricting the controllers to only manage a specific namespace, and making sure your ingress resource specifies an ingress class by the ingress-class annotation and/or the ingressClass block.

Blender Fox
  • 4,442
  • 2
  • 17
  • 30
  • There is no ingressClass.. is it normal? -- kubectl.exe get IngressClass No resources found. --edit: I re-deploy ingress-nginx and right now there is a ingressClass.. but I still dont understand, how can I manage internal and external ingress objects – thargalin Jul 18 '22 at 07:37
  • An ingress controller should normally setup its own `ingressClass` resource if your cluster supports it. If you're running an old version of kubernetes then you might not have support for the ingressClass block, and instead, you should use the annotation `kubernetes.io/ingress.class: {ingress-class-name}` – Blender Fox Jul 18 '22 at 07:43
  • This setup working on AKS and version 1.22.6.. How can I create a new ingressClass? I searched on internet but most article using helm. I don't know helm. How can I create with yaml? – thargalin Jul 18 '22 at 07:55
  • https://kubernetes.io/docs/concepts/services-networking/ingress/ is a good place to start – Blender Fox Jul 18 '22 at 08:31
1

I believe I had made the same mistake and managed to find the solution after reading through this post. If you have a similar kubernetes behavioury you might wanna try and take a look at: https://stackoverflow.com/a/76230364/7753261

Florian7843
  • 303
  • 3
  • 12
0

my problem is about the IngressClass and Ingress installaion section. Unfortunately I had to use yaml while installing, but then I had to uninstall and reinstall, so there was a problem.

Lesson learned from this process is use helm or be careful when using yaml.

thargalin
  • 123
  • 1
  • 2
  • 16