Kuberenetes ExternalName service forwarding request to service outside cluster

Question

We have a requirement to forward the request to service outside of cluster.

/ -> some service outside cluster (someapi.com)

/api -> service inside cluster

When I try to hit the https://someapi.com/health it gives me proper response but not through ingress.

Ingress

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: custom-ingress
  annotations:
    kubernetes.io/ingress.class: haproxy
status:
  loadBalancer: {}
spec:
  tls:
    - hosts:
        - mytenant.com
      secretName: tenant-secret
  rules:
    - host: mytenant.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: external-service
                port:
                  number: 80

Service

apiVersion: v1
kind: Service
metadata:
  name: external-service
status:
  loadBalancer: {}
spec:
  type: ExternalName
  sessionAffinity: None
  externalName: someapi.com

curl -ikv https://mytenant.com/health is giving me

503 Service Unavailable

No server is available to handle this request.

Connection #0 to host mytenant.com left intact

I tried nslookup and it does evaluate to ip

/usr/src/app # nslookup external-service
Server:         901.63.1.11
Address:        901.63.1.11:53

external-service.default.svc.cluster.local    canonical name = someapi.com
someapi.com        canonical name = proxy-aws-can-55.elb.eu-central-1.amazonaws.com
Name:   proxy-aws-can-55.elb.eu-central-1.amazonaws.com
Address: 92.220.220.137
Name:   proxy-aws-can-55.elb.eu-central-1.amazonaws.com
Address: 33.43.161.163
Name:   proxy-aws-can-55.elb.eu-central-1.amazonaws.com
Address: 98.200.178.250

external-service.default.svc.cluster.local    canonical name = someapi.com
someapi.com        canonical name = proxy-aws-can-55.elb.eu-central-1.amazonaws.com

When I changed the external-service port to 80 (also tried changing target port of service to 443)

spec:
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  type: ExternalName
  sessionAffinity: None
  externalName: someapi.com

It keeps on looping with 301

< HTTP/2 301
< content-length: 0
< location: https://mytenant.com/health
< strict-transport-security: max-age=15768000

(With same setup if I just change the externalName to httpbin.org it works fine.)

When I changed ingress (port) and service (port and targetPort) to 443, I am getting

REFUSED_STREAM, retrying a fresh connect
Connection died, tried 5 times before giving up
Closing connection 5
curl: (56) Connection died, tried 5 times before giving up

I also tried setting host header mentioned here, https://www.haproxy.com/documentation/kubernetes/latest/configuration/ingress/#set-host but no luck still 301.

Please help me understand how I should make it work. many thanks!

Since you get a 503 at your service and it worked on public (httpbin) site, you have to provide more information about your service. Why nslookup resolves it as such a weird address `901.63.1.11`? — Michal, Jan 27 '23 at 02:52
Thanks Michal for your help. I changed the actual data but the nslookup does evaluate to correct ip. service someapi.com is deployed in cloud foundry and is looking for valid host header. — Jayesh, Jan 27 '23 at 03:57
Thanks Michal. I got it working after changing the ingress (port) to 443 and also the service (port and targetPort) to 443. also I added the annotation (ingress.kubernetes.io/backend-protocol: h1-ssl) in my ingress after referring this post https://github.com/kubernetes/ingress-nginx/issues/6498 — Jayesh, Jan 28 '23 at 01:19
@Michal I don't understand the overall flow. client to haproxy ingress controller (with ssl termination). if i change my ingress (port to 80) and service (port to 80, targetPort to 443) it doesn't work and says 503 Service Unavailable. not sure what is the relevance of ingress (port) and service (port and target port) in case of service of type ExternalName. wondering why ingress port and service port needs to be 443 to make this work? — Jayesh, Jan 28 '23 at 01:28
One query you are configuring an external outbound service at ingress? Ingress is used for incoming traffic it doesn’t require any configuration for outbound traffic unless an egress is configured and it’s needs to be routed through egress — Nataraj Medayhal, Jan 29 '23 at 08:47

score 1 · Answer 1 · answered Jan 31 '23 at 02:51

I got the working configuration, I changed the ingress (port) and service (port/targetPort) to 443. Also, added annotation ingress.kubernetes.io/backend-protocol: h1-ssl on ingress.

I believe I was getting 301 because the upstream service was expecting https request and after adding the backend-protocol annotation, after ssl termination at HA Proxy controller, the new call that initiated was https and that fulfilled the request. Also, I think value for Service targetPort doesn't matter in case of ExternalName service.

Ingress

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: custom-ingress
  annotations:
    ingress.kubernetes.io/backend-protocol: h1-ssl
    kubernetes.io/ingress.class: haproxy
status:
  loadBalancer: {}
spec:
  tls:
    - hosts:
        - mytenant.com
      secretName: tenant-secret
  rules:
    - host: mytenant.com
      http:
        paths:
          - path: /
            pathType: Prefix
            backend:
              service:
                name: external-service
                port:
                  number: 443

Service

apiVersion: v1
kind: Service
metadata:
  name: external-service
status:
  loadBalancer: {}
spec:
  ports:
    - protocol: TCP
      port: 443
      targetPort: 443
  type: ExternalName
  sessionAffinity: None
  externalName: someapi.com

Kuberenetes ExternalName service forwarding request to service outside cluster

1 Answers1

Linked