11

I have created an ingress controller using Helm with default configuration

default        nginx-ingress-controller        LoadBalancer   10.0.182.128   xx.xxx.xx.90     80:32485/TCP,443:31756/TCP   62m
default        nginx-ingress-default-backend   ClusterIP      10.0.12.39     <none>           80/TCP                       62m

using Helm:

helm install nginx-ingress stable/nginx-ingress \         
--set controller.replicaCount=2 \     
--set controller.nodeSelector."beta\.kubernetes\.io/os"=linux \
--set defaultBackend.nodeSelector."beta\.kubernetes\.io/os"=linux \
--set controller.service.loadBalancerIP="Created static IP" \
--set controller.service.annotations."service\.beta\.kubernetes\.io/azure-dns-label-name"="XXX-aks-ingress"

this ingress is running in the default namespace.

Now, I wanted to add a second ingress controller, from the official doc I have specific Ingress class

helm install nginx-ingress stable/nginx-ingress \     
--namespace ingress-nginx-devices \ #I create this namespace first 
--set controller.ingressClass="nginx-devices" \   # custom class to use for different ingress resources  
--set controller.replicaCount=2 \     
--set controller.nodeSelector."beta\.kubernetes\.io/os"=linux \
--set defaultBackend.nodeSelector."beta\.kubernetes\.io/os"=linux \
--set controller.service.loadBalancerIP="A second static Ip address created before" \
--set controller.service.annotations."service\.beta\.kubernetes\.io/azure-dns-label-name"="serviceIot-aks-ingress-iot"

but I keep getting this error:

Error: rendered manifests contain a resource that already exists. Unable to continue with install: ClusterRole "nginx-ingress" in namespace "" exists and cannot be imported into the current release: invalid ownership metadata; annotation validation error: key "meta.helm.sh/release-namespace" must equal "ingress-nginx-devices": current value is "default"

What could be wrong here ? Any help is appreciated :)

ikenahim
  • 331
  • 1
  • 3
  • 15

3 Answers3

13

For my case the issue was with ingressclass existed already. I have simply deleted the ingresclass and it worked like a charm.

# kubectl get ingressclass --all-namespaces

This will return the name of already existing ingressclass. For my case, it was nginx. Delete that ingressclass.

# kubectl delete ingressclass nginx --all-namespaces

Verify that ingressclass is deleted

# kubectl get ingressclass --all-namespaces
No resources found

Rerun the helm update command should work.

You can also run multiple ingrss controller in parallel with new ingressClassName & ingressClassResourceName. First of all get list of all existing class name.

kubectl get ingressclass --all-namespaces
NAME        CONTROLLER             PARAMETERS   AGE
nginx       k8s.io/ingress-nginx   <none>       203d

Create a new ingress controller with unique className with this command.

helm install nginx-ingress stable/nginx-ingress \     
--namespace ingress-nginx-devices \ #I create this namespace first 
--set controller.ingressClass="nginx-devices" \   # custom class to use for different ingress resources 
--set controller.ingressClassResource.name="nginx-devices" # custom classResourceName   
--set controller.replicaCount=2 \     
--set controller.nodeSelector."beta\.kubernetes\.io/os"=linux \
--set defaultBackend.nodeSelector."beta\.kubernetes\.io/os"=linux \
--set controller.service.loadBalancerIP="A second static Ip address created before" \
--set controller.service.annotations."service\.beta\.kubernetes\.io/azure-dns-label-name"="serviceIot-aks-ingress-iot"
iamattiq1991
  • 746
  • 9
  • 11
3

Update :

Controller-value of the controller that is processing this ingressClass

With recent update you might have to use the controller.ingressClassByName

Official doc : https://github.com/kubernetes/ingress-nginx/tree/main/charts/ingress-nginx#values

helm install nginx-ingress-devices stable/nginx-ingress \     
    --namespace ingress-nginx-devices \ #I create this namespace first 
    --set controller.ingressClass="nginx-devices" \   # custom class to use for different ingress resources  
    --set controller.ingressClassResource.name="nginx-devices" \
    --set controller.replicaCount=2 \     
    --set controller.nodeSelector."beta\.kubernetes\.io/os"=linux \
    --set defaultBackend.nodeSelector."beta\.kubernetes\.io/os"=linux \
    --set controller.service.loadBalancerIP="A second static Ip address created before" \
    --set controller.service.annotations."service\.beta\.kubernetes\.io/azure-dns-label-name"="serviceIot-aks-ingress-iot"

Old answer

you can try, what we are changing is name : nginx-ingress-devices instead of nginx-ingress

helm install nginx-ingress-devices stable/nginx-ingress \     
--namespace ingress-nginx-devices \ #I create this namespace first 
--set controller.ingressClass="nginx-devices" \   # custom class to use for different ingress resources  
--set controller.replicaCount=2 \     
--set controller.nodeSelector."beta\.kubernetes\.io/os"=linux \
--set defaultBackend.nodeSelector."beta\.kubernetes\.io/os"=linux \
--set controller.service.loadBalancerIP="A second static Ip address created before" \
--set controller.service.annotations."service\.beta\.kubernetes\.io/azure-dns-label-name"="serviceIot-aks-ingress-iot"

error you are getting is due to already there is cluster role with same name : nginx-ingress due to that you are getting the error.

ClusterRoleBindings grant a user, group, or service account a ClusterRole’s power across the entire cluster.

You can get the reference file here : https://github.com/helm/charts/blob/master/stable/nginx-ingress/templates/clusterrole.yaml

Harsh Manvar
  • 27,020
  • 6
  • 48
  • 102
  • One remark is that the moment I created a clusterIssuer to get TLS certificate for the new ingress its overwrite the TLS certificate I had for the first controller, is there a way to fix this ? – ikenahim Apr 20 '21 at 10:50
  • i think TLS cert get stored in secret maybe ? you can change the name of secret – Harsh Manvar Apr 20 '21 at 10:52
  • 1
    Yes it is but the clusterissuer has no place to store it `# k8s/cluster-issuer.yaml apiVersion: cert-manager.io/v1alpha2 kind: ClusterIssuer metadata: # name: letsencrypt-staging name: letsencrypt-prod-devices spec: acme: email: xxx # server: https://acme-staging-v02.api.letsencrypt.org/directory server: https://acme-v02.api.letsencrypt.org/directory privateKeySecretRef: # name: letsencrypt-staging name: letsencrypt-prod-devices solvers: - http01: ingress: class: nginx-devices` – ikenahim Apr 20 '21 at 10:54
  • the ingress resource rules use the secret `secretName: tls-secret` but im not sure where its been created, I only use ClusterIssuer and Ingress rule – ikenahim Apr 20 '21 at 10:54
  • you can use the `kubectl get secrets` and check it or with respective namespace using `kubectl get secrets -n ingress-nginx-devices` clusterissuer automatic create the secret you can check it – Harsh Manvar Apr 20 '21 at 11:00
  • I cant find it, tls-secret is only created for namespace default, If I create a ClusterIssuer and ingress rules in `ingress-nginx-devices` knowing that all my services are in default namespace, do I get a new tls-secret in the namespace`ingress-nginx-devices` and also with correct routing ? – ikenahim Apr 20 '21 at 11:07
  • sorry not gettig, ideally you have to cluster issuer and ingress in the `namespace` in which your service is running simple. cluster issuer will create auto secret in that same namespace you can give different secret name for each ingress or cluster issuer and it will create it automatically. – Harsh Manvar Apr 20 '21 at 11:28
  • I have created a separate question about this, if you can check it out [https://stackoverflow.com/questions/67178086/two-tls-certificate-for-two-nginx-ingress-controller-azure-k8s-cluster] Thanks a lot :) – ikenahim Apr 20 '21 at 11:33
  • Nowadays the correct answer is the one provided by @hargut: https://stackoverflow.com/a/72981072/107102 – jmservera Jul 05 '23 at 22:37
0

With recent releases "controller.ingressClassResource.name" needs to get a different name when trying to deploy a second ingress.

As shown in the list of keys now there's a new value that you need to set:

Key Type Default Description
... ...
controller.ingressClass string "nginx" For backwards compatibility with ingress.class annotation, use ingressClass. Algorithm is as follows, first ingressClassName is considered, if not present, controller looks for ingress.class annotation
controller.ingressClassByName bool false Process IngressClass per name (additionally as per spec.controller).
controller.ingressClassResource.controllerValue string "k8s.io/ingress-nginx" Controller-value of the controller that is processing this ingressClass
controller.ingressClassResource.default bool false Is this the default ingressClass for the cluster
controller.ingressClassResource.enabled bool true Is this ingressClass enabled or not
controller.ingressClassResource.name string "nginx" Name of the ingressClass

So, for example, for creating a secondary ingress for private access you can run this Helm command:

helm install nginx-ingress-private ingress-nginx/ingress-nginx \
 --create-namespace \
 --namespace ingress-private \
 --set controller.ingressClass="nginx-private" \
 --set controller.ingressClassResource.name="nginx-private" \
 --set controller.service.annotations."service\.beta\.kubernetes\.io/azure-load-balancer-health-probe-request-path"=/healthz \
 --set controller.service.annotations."service\.beta\.kubernetes\.io/azure-load-balancer-internal"="true"
jmservera
  • 6,454
  • 2
  • 32
  • 45
hargut
  • 109
  • 1
  • 2
    Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Jul 17 '22 at 00:53