308 Redirect Loop with ExternalName Service Using Ingress-NGINX

Question

I'm using ingress-nginx-controller (0.32.0) and am attempting to point an ExternalName service at a URL and yet it’s stuck in a loop of 308 Redirects. I've seen plenty of issues out there and I figure there’s just one thing off with my configuration. Is there something really small that I'm missing here?

ConfigMap for NGINX Configuration:

kind: ConfigMap
apiVersion: v1
metadata:
  name: nginx-configuration
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
data:
  use-proxy-protocol: "true"
  use-forwarded-headers: "true"
  proxy-real-ip-cidr: "0.0.0.0/0" # restrict this to the IP addresses of ELB
  proxy-read-timeout: "3600"
  proxy-send-timeout: "3600"
  backend-protocol: "HTTPS"
  ssl-redirect: "false"
  http-snippet: |
    map true $pass_access_scheme {
      default "https";
    }
    map true $pass_port {
      default 443;
    }
    server {
      listen 8080 proxy_protocol;
      return 308 https://$host$request_uri;
    }

NGINX Service:

kind: Service
apiVersion: v1
metadata:
  name: ingress-nginx
  namespace: ingress-nginx
  labels:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: "XXX"
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: "tcp"
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "https"
    service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "3600"
    service.beta.kubernetes.io/aws-load-balancer-proxy-protocol: "*"
spec:
  type: LoadBalancer
  selector:
    app.kubernetes.io/name: ingress-nginx
    app.kubernetes.io/part-of: ingress-nginx
  ports:
    - name: http
      port: 80
      targetPort: 8080
    - name: https
      port: 443
      targetPort: http

Ingress Definition:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-members-portal
  namespace: dev
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/use-regex: "true"
spec:
  rules:
  - host: foo-111.dev.bar.com
    http:
      paths:
      - path: /login
        backend:
          serviceName: foo-service
          servicePort: 80

ExternalName Service:

apiVersion: v1
kind: Service
metadata:
  name: foo-service
spec:
  type: ExternalName
  externalName: foo.staging.bar.com
selector:
  app: foo

EDIT

I figured it out! I wanted to point to a service in another namespace, so I changed the ExternalName service to this:

apiVersion: v1
kind: Service
metadata:
  name: foo-service
spec:
  type: ExternalName
  externalName: foo-service.staging.svc.cluster.local
ports:
- port: 80
  protocol: TCP
  targetPort: 80
selector:
  app: foo

Answer 1

I believe the issue you're seeing is due to the fact that your external service isn't working as you think it is. In your ingress definition, you are defining the service to utilize port 80 on your foo-service. In theory, this would redirect you back to your ingress controller's ELB, redirect your request to the https://foo.staging.bar.com address, and move on.

However, external services don't really work that way. Essentially, all externalName will do is run a DNS check with KubeDNS/CoreDNS, and return the CNAME information on that request. It doesn't handle redirects of any kind.

For example, in this case, foo.staging.bar.com:80 would return foo.staging.bar.com:443. You are directing the request for that site to port 80, which in itself directs the request to port 8080 in the ingress controller, which then redirects that request back out to the ELB's port 443. That redirect logic doesn't coexist with the external service.

The problem here, then, is that your app will essentially then try to do this:

http://foo-service:80 --> http://foo.staging.bar.com:80/login --> https://foo.staging.bar.com:443

My expectation on this is that you never actually reach the third step. Why? Well because foo-service:80 is not directing you to port 443, first of all, but second of all...all coreDNS is doing in the backend is running a DNS check against the foo-service's external name, which is foo.staging.bar.com. It's not handling any kind of redirection. So depending on how your host from your app is returned, and handled, your app may never actually get to that site and port. So rather than reach that site, you just have your app keep looping back to http://foo-service:80 for those requests, which will always result in a 308 loopback.

The key here is that foo-service is the host header being sent to the NGINX Ingress controller, not foo.staging.bar.com. So on a redirect to 443, my expectation is that all that is happening, then, is you are hitting foo-service, and any redirects are being improperly sent back to foo-service:80.

A good way to test this is to run curl -L -v http://foo-service:80 and see what happens. That will follow all redirects from that service, and provide you context as to how your ingress controller is handling those requests.

It's really hard to give more information, as I don't have access to your setup directly. However, if you know that your app is always going to be hitting port 443, it would probably be a good fix, in this case, to change your ingress and service to look something like this:

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: ingress-members-portal
  namespace: dev
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/use-regex: "true"
spec:
  rules:
  - host: foo-111.dev.bar.com
    http:
      paths:
      - path: /login
        backend:
          serviceName: foo-service
          servicePort: 443
---
apiVersion: v1
kind: Service
metadata:
  name: foo-service
spec:
  type: ExternalName
  externalName: foo.staging.bar.com

That should ensure you don't have any kind of https redirect. Unfortunately, this may also cause issues with ssl validation, but that would be another issue all together. The last piece of I can recommend is to possibly just use foo.staging.bar.com itself, rather than an external service in this case.

For more information, see: https://kubernetes.io/docs/concepts/services-networking/service/#externalname

Hope this helps!