0

I have an existing LoadBalancer which is configured to listen on 4443 and its mTLS TCP based backend. Now i am trying to convert this existing Load balancer setup to work via Kubernetes nginx ingress controller. Backends configured are listening on port 19000.

My existing LB setup looks like this with MTLS example of how it works.

curl -v --cacert ./certs_test/ca.crt --key ./certs_test/server.key --cert ./certs_test/server.crt -k https://10.66.49.164:4443/api/testtools/private/v1/test/testGetTenants

With nginx ingress controller i have created a private LB override default 443 port to 4443 in args section like below.

            - /nginx-ingress-controller
            - --publish-service=$(POD_NAMESPACE)/ingress-chart
            - --election-id=ingress-controller-leader
            - --ingress-class=nginx
            - --configmap=$(POD_NAMESPACE)/ingress-chart
            - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
            - --https-port=4443
            - --default-ssl-certificate=$(POD_NAMESPACE)/tls-secret
            - --enable-ssl-passthrough

I want to terminate MTLS at ingress level so that the authentication will work as expected. Below is an example of the setup which i currently have.

Clusterip service yaml

apiVersion: v1
kind: Service
metadata:
  name: testtools-service
spec:
  type: ClusterIP
  ports:
  - port: 19000
    protocol: TCP
    targetPort: 19000
    name: https-testtools
  selector:
    app: testtools

Deployment yaml spec

apiVersion: apps/v1
kind: Deployment
metadata:
  name: testtools-deployment
spec:
  selector:
    matchLabels:
      app: testtools
  replicas: 3
  template:
    metadata:
      labels:
        app: testtools
    spec:
      containers:
      - name: testtools
        image: tst-testtools-service-np:214
        livenessProbe:
          httpGet:
            path: /api/testtools/private/health
            port: 19001
            scheme: HTTPS
          initialDelaySeconds: 300
        readinesProbe:
          httpGet:
            path: /api/testtools/private/health
            port: 19001
            scheme: HTTPS
          initialDelaySeconds: 120
        ports:
        - containerPort: 19000
          name: https-testtools
        volumeMounts:
        - mountPath: /test/logs
          name: test-volume
        - mountPath: /etc/pki
          name: etc-service
        - mountPath: /etc/environment
          name: etc-env
        - mountPath: /etc/availability-domain
          name: etc-ad
        - mountPath: /etc/identity-realm
          name: etc-idr
        - mountPath: /etc/region
          name: etc-region
        - mountPath: /etc/fault-domain
          name: etc-fd
        - mountPath: /etc/hostclass
          name: etc-hc
        - mountPath: /etc/physical-availability-domain
          name: etc-pad
        resources:
          requests:
            memory: "8000Mi"
      volumes:
      - name: test-volume
        hostPath:
          path: /test/logs
          type: Directory
      - name: etc-service
        hostPath:
          path: /etc/pki
          type: Directory
      - name: etc-env
        hostPath:
          path: /etc/environment
          type: File
      - name: etc-ad
        hostPath:
          path: /etc/availability-domain
          type: File
      - name: etc-idr
        hostPath:
          path: /etc/identity-realm
          type: File
      - name: etc-region
        hostPath:
          path: /etc/region
          type: File
      - name: etc-fd
        hostPath:
          path: /etc/fault-domain
          type: File
      - name: etc-hc
        hostPath:
          path: /etc/hostclass
          type: File
      - name: etc-pad
        hostPath:
          path: /etc/physical-availability-domain
          type: File
      dnsPolicy: Default

My ingress spec

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: testtools-ingress
  annotations:
    kubernetes.io/ingress.class: "nginx"
    nginx.ingress.kubernetes.io/ssl-passthrough: "true"
    #nginx.ingress.kubernetes.io/use-regex: "true"
    #nginx.ingress.kubernetes.io/rewrite-target: /$1
    nginx.ingress.kubernetes.io/backend-protocol: "HTTPS"
    nginx.ingress.kubernetes.io/secure-backends: "true"
    #nginx.ingress.kubernetes.io/auth-tls-verify-client: "on"
    #nginx.ingress.kubernetes.io/auth-tls-secret: "default/tls-new-cert"
    #nginx.ingress.kubernetes.io/auth-tls-verify-depth: "3"
    #nginx.ingress.kubernetes.io/auth-tls-pass-certificate-to-upstream: "true"
    nginx.ingress.kubernetes.io/auth-tls-secret: default/my-certs
    nginx.ingress.kubernetes.io/auth-tls-verify-client: "on"
    nginx.ingress.kubernetes.io/auth-tls-verify-depth: "3"
    nginx.ingress.kubernetes.io/proxy-ssl-name: ingress-nginx-controller
    nginx.ingress.kubernetes.io/proxy-ssl-secret: default/my-certs
    nginx.ingress.kubernetes.io/proxy-ssl-verify: "on"
    nginx.ingress.kubernetes.io/proxy-ssl-verify-depth: "5"
spec:
  tls:
  - secretName: my-certs
  rules:
  - http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: testtools-service
            port:
              number: 19000

what i am trying to achieve is do passthrough at load balancer level and terminate at ingress to ensure mtls is working as expected and api call to succeed. Want to know if my ingress setup is correct. Would like to know if this achievable using k8s ingress controller with LB as service this is straight forward but to reduce creating too many LB i was trying this setup. Termination at ingress controller is working fine but further calls just dont provide any response. Your inputs on this would help.

$ curl -v --cacert ./certs_test/ca.crt --key ./certs_test/server.key --cert ./certs_test/server.crt -k https://10.66.48.120:4443/api/testtools/private/v1/test/testGetTenants
* About to connect() to 10.66.48.120 port 4443 (#0)
*   Trying 10.66.48.120...
* Connected to 10.66.48.120 (10.66.48.120) port 4443 (#0)
* Initializing NSS with certpath: sql:/etc/pki/nssdb
*   CAfile: ./certs_test/ca.crt
  CApath: none
* skipping SSL peer certificate verification
* SSL connection using TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
* Server certificate:
*   subject: O=Acme Co,CN=Kubernetes Ingress Controller Fake Certificate
*   start date: Feb 23 18:03:52 2022 GMT
*   expire date: Feb 23 18:03:52 2023 GMT
*   common name: Kubernetes Ingress Controller Fake Certificate
*   issuer: O=Acme Co,CN=Kubernetes Ingress Controller Fake Certificate
> GET /api/prodtools/private/v1/common/getTenants HTTP/1.1
> User-Agent: curl/7.29.0
> Host: 10.66.48.120:4443
> Accept: */*
>
* Connection #0 to host 10.66.48.120 left intact
P
$
vinodh
  • 167
  • 2
  • 8
  • Do I understand correctly that you have a LoadBalancer that directs traffic to the ingress? Could you please provide more information about `existing Load balancer setup`? What version of Kubernetes do you have? Do you have Kubernetes on bare-metal or a Managed Kubernetes Service like GKE, AKS, EKS? Are you using [kubernetes/ingress-nginx](https://github.com/kubernetes/ingress-nginx) as Ingress Controller? – Andrew Skorkin Mar 07 '22 at 13:01
  • 1
    Additionally, I think you may be interested in this [answer](https://stackoverflow.com/a/63027100/14801225) and this [blog](https://awkwardferny.medium.com/configuring-certificate-based-mutual-authentication-with-kubernetes-ingress-nginx-20e7e38fdfca). – Andrew Skorkin Mar 07 '22 at 13:01
  • I have 1.19.5 version of k8s engine in my current setup. I am using OKE managed service in oci. existing LB has TCP Listener on port 4443 and directs packets to backend and uses mTLS authentication. – vinodh Mar 07 '22 at 23:31
  • Did you look at the links from my previous comment? It describes step by step how to configure mTLS. Do you have any updates based on this information? – Andrew Skorkin Mar 08 '22 at 12:04
  • yes that is helpful i have one more doubt is it possible to have ingress controller work just as a pass thru and do mTLS termination at ingress level. – vinodh Mar 08 '22 at 21:54
  • As we can find in [the documentation](https://kubernetes.github.io/ingress-nginx/user-guide/nginx-configuration/annotations/#ssl-passthrough), this is not possible because using SSL Passthrough invalidates all the other annotations set on an Ingress object. – Andrew Skorkin Mar 10 '22 at 11:18
  • Hi @vinodh! Any progress on this issue? – Andrew Skorkin Mar 17 '22 at 08:20

0 Answers0