Path based rules giving 404 for AKS ingress controller

Question

I have two ClusterIP services on AKS and an ingress controller ingress-nginx-4.6.1 when I try the following config to route the traffic to each service I get 404.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: spark-ingress
  annotations:
    nginx.ingress.kubernetes.io/backend-protocol: HTTPS
    nginx.ingress.kubernetes.io/rewrite-target: /$2
    nginx.ingress.kubernetes.io/use-regex: "true"
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - spark.company.net
    secretName: tls-secret
  rules:
  - host: spark.company.net
    http:
      paths:
      - path: /service1(/|$)(.*)
        pathType: Prefix
        backend:
          service:
            name: service-1
            port:
              number: 443
      - path: /service2(/|$)(.*)
        pathType: Prefix
        backend:
          service:
            name: service-2
            port:
              number: 443

the output of kubectl describe ing is:

Name:             spark-ingress
Labels:           <none>
Namespace:        apache-spark
Address:          10.0.0.0
Default backend:  default-http-backend:80 (<error: endpoints "default-http-backend" not found>)
TLS:
  tls-secret terminates spark.company.net
Rules:
  Host                  Path  Backends
  ----                  ----  --------
  spark.company.net
                        /service1(/|$)(.*) service-1:443 (100.64.101.8:4440)
                        /service1(/|$)(.*) service-2:443 (100.64.101.0:4440)
Annotations:            
                        nginx.ingress.kubernetes.io/backend-protocol: HTTPS
                        nginx.ingress.kubernetes.io/rewrite-target: /$2
                        nginx.ingress.kubernetes.io/use-regex: true
Events:
  Type    Reason  Age                    From                      Message
  ----    ------  ----                   ----                      -------
  Normal  Sync    2m9s (x3165 over 26h)  nginx-ingress-controller  Scheduled for sync
  Normal  Sync    2m9s (x3165 over 26h)  nginx-ingress-controller  Scheduled for sync

It's worth noting that the backend application is open source spark. so the default behavior when hitting the app is to 302 redirect to /jobs/. I would also like to point out that when I define the ing for one service (does not matter which) on the path / it works just fine. I need to do this for multiple services. Any help would be appreciated.

score 0 · Accepted Answer · answered Jun 07 '23 at 15:06

The issue was that my backend application (Apache Spark) was sending redirects to the client. so the client is now requesting http://baseurl/resource instead of http://baseurl/service1/resource

The fix was to add the following properties to the spark-submit

 --conf spark.ui.proxyBase=/service1\
 --conf spark.ui.proxyRedirectUri=localhost/service1\

For all redirects spark ui will now use service1 as base which inginx has a rule for and knows how to handle.

Path based rules giving 404 for AKS ingress controller

1 Answers1