Custom response for Ingress-Nginx External Authentication

Question

I have deployed my Kubernetes cluster on EKS. I have an ingress-nginx which is exposed via load balancer to route traffic to different services. In ingress-nginx first request goes to auth service for authentication and if it is a valid request then I allow it to move forward. This is done using ingress-nginx annotation nginx.ingress.kubernetes.io/auth-url. Auth service is developed using FastAPI. In case of 401 response from fastAPI look like this FASTAPI

But when I use ingress-nginx the response look like this INGRESS_NGINX

Is there a way to get JSON respone from Ingress-nginx? Ingress File

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: ingress-service
  annotations:
    kubernetes.io/ingress.class: 'nginx'
    nginx.ingress.kubernetes.io/use-regex: 'true'
    nginx.ingress.kubernetes.io/rewrite-target: /$1
    nginx.ingress.kubernetes.io/auth-response-headers: item_id
    nginx.ingress.kubernetes.io/auth-method: POST
    nginx.ingress.kubernetes.io/auth-url: http://pth-auth.default.svc.cluster.local:8000/item/1
    # UPDATE THIS LINE ABOVE
spec:
  rules:
    - http:
        paths:
          - path: /?(.*)
            # UPDATE THIS LINE ABOVE
            backend:
              serviceName: client-cluster-ip-service
              servicePort: 3000
          - path: /api/?(.*)
            # UPDATE THIS LINE ABOVE
            backend:
              serviceName: server-cluster-ip-service
              servicePort: 5000
          - path: /pth-auth/?(.*)
            # UPDATE THIS LINE ABOVE
            backend:
              serviceName: pth-auth
              servicePort: 8000

Please take a look at this https://stackoverflow.com/questions/62770975/how-to-perform-custom-authentication-with-kubernetes-ingress https://stackoverflow.com/questions/58997958/oauth2-proxy-authentication-calls-slow-on-kubernetes-cluster-with-auth-annotatio Can you add tls section to ingress yaml file and and access url via https - change it in annotation. — Malgorzata, Jan 29 '21 at 14:10

Tereza Tomcova · Answer 1 · 2021-10-06T14:43:31.273

Here's a solution that worked for me. It allows the auth service to return a custom error message for each request.

The caveat is that because nginx can't access auth response body, the pth-auth service needs to put the data in Pth-Auth-Error header (base64-encoded).

This example handles 401, 500, and a special case when pth-auth service is unavailable.

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: ingress-service
  annotations:
    kubernetes.io/ingress.class: 'nginx'
    nginx.ingress.kubernetes.io/use-regex: 'true'
    nginx.ingress.kubernetes.io/rewrite-target: /$1
    nginx.ingress.kubernetes.io/auth-response-headers: item_id
    nginx.ingress.kubernetes.io/auth-method: POST
    nginx.ingress.kubernetes.io/auth-url: http://pth-auth.default.svc.cluster.local:8000/item/1
    # UPDATE THIS LINE ABOVE
    nginx.ingress.kubernetes.io/configuration-snippet: |
      # Redirect auth errors to custom named locations
      error_page 401 = @ingress_service_custom_error_401;
      error_page 500 = @ingress_service_custom_error_500;

      # Grab data from auth error response
      auth_request_set $pth_auth_error $upstream_http_pth_auth_error;
      auth_request_set $pth_auth_error_content_type $upstream_http_content_type;
      auth_request_set $pth_auth_status $upstream_status;

    nginx.ingress.kubernetes.io/server-snippet: |
      location @ingress_service_custom_error_401 {
        internal;

        # Decode auth response header
        set_decode_base64 $pth_auth_error_decoded $pth_auth_error;

        # Return the error from pth-auth service if any
        if ($pth_auth_error_decoded != ""){
          add_header Content-Type $pth_auth_error_content_type always;
          return 401 $pth_auth_error_decoded;
        }

        # Fall back to default nginx response
        return 401;
      }

      location @ingress_service_custom_error_500 {
        internal;

        # Decode auth response header
        set_decode_base64 $pth_auth_error_decoded $pth_auth_error;

        # Return the error from pth-auth service if any
        if ($pth_auth_error_decoded != ""){
          add_header Content-Type $pth_auth_error_content_type always;
          return 500 $pth_auth_error_decoded;
        }

        # Return a hardcoded error in case no pth-auth pods are available
        if ($pth_auth_status = 503){
          add_header Content-Type application/json always;
          return 503 "{\"msg\":\"pth-auth service is unavailable\"}";
        }

        # Fall back to default nginx response
        return 500;
      }
spec:
  rules:
    - http:
        paths:
          - path: /?(.*)
            # UPDATE THIS LINE ABOVE
            backend:
              serviceName: client-cluster-ip-service
              servicePort: 3000
          - path: /api/?(.*)
            # UPDATE THIS LINE ABOVE
            backend:
              serviceName: server-cluster-ip-service
              servicePort: 5000
          - path: /pth-auth/?(.*)
            # UPDATE THIS LINE ABOVE
            backend:
              serviceName: pth-auth
              servicePort: 8000

Inspired by: https://stackoverflow.com/a/31485557/99237

Troubleshooting tips:

Here's the template nginx ingress uses when transforming the ingress annotations into nginx config file.
Connect to the ingress controller pod and look at /etc/nginx/nginx.conf to view the generated nginx config.

score 1 · Accepted Answer · answered Jan 30 '21 at 15:22

This worked for me, took reference from here https://github.com/kubernetes/ingress-nginx/issues/2292

apiVersion: networking.k8s.io/v1beta1
kind: Ingress
metadata:
  name: ingress-service
  annotations:
    kubernetes.io/ingress.class: 'nginx'
    nginx.ingress.kubernetes.io/use-regex: 'true'
    nginx.ingress.kubernetes.io/rewrite-target: /$1
    nginx.ingress.kubernetes.io/auth-response-headers: item_id
    nginx.ingress.kubernetes.io/auth-method: POST
    nginx.ingress.kubernetes.io/auth-url: http://pth-auth.default.svc.cluster.local:8000/items/1
    nginx.ingress.kubernetes.io/server-snippet: |
      location = /error/401 {
        proxy_method POST;
        proxy_pass http://pth-auth.default.svc.cluster.local:8000/error/401;
      }
      location = /error/403 {
        proxy_method POST;
        proxy_pass http://pth-auth.default.svc.cluster.local:8000/error/403;
      }
    nginx.ingress.kubernetes.io/configuration-snippet: |
      error_page 401 /error/401;
      error_page 403 /error/403;
    # UPDATE THIS LINE ABOVE
spec:
  rules:
    - http:
        paths:
          - path: /?(.*)
            # UPDATE THIS LINE ABOVE
            backend:
              serviceName: client-cluster-ip-service
              servicePort: 3000
          - path: /api/?(.*)
            # UPDATE THIS LINE ABOVE
            backend:
              serviceName: server-cluster-ip-service
              servicePort: 5000
          - path: /pth-auth/?(.*)
            # UPDATE THIS LINE ABOVE
            backend:
              serviceName: pth-auth
              servicePort: 8000

You just need to tell nginx in case of error route traffic to this location and their your function will handle specific errors. In my case, function is error/{error_code}.

Custom response for Ingress-Nginx External Authentication

2 Answers2

Troubleshooting tips: