1

I deployed an Istio Service Mesh, and I use its gateway controller for ingress. I setup cert-manager which passes ssl certificates to the gateways. With self-signed certificates this setup works fine, but when using letsencrypt, I have a conflict between cert-manager's automated temporary ingress, and the istio gateway.

Here's the resulting setup, for httpbin:

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  annotations:
    meta.helm.sh/release-name: httpbin-ingress
    meta.helm.sh/release-namespace: httpbin
  creationTimestamp: "2022-10-13T08:07:33Z"
  generation: 1
  labels:
    app.kubernetes.io/managed-by: Helm
  name: httpbin-ingress
  namespace: istio-ingress
  resourceVersion: "5243"
  uid: d4087649-2609-40c0-8d4a-55b9a420fda9
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - httpbin.example.com
    port:
      name: http
      number: 80
      protocol: HTTP
    tls:
      httpsRedirect: true
  - hosts:
    - httpbin.example.com
    port:
      name: https
      number: 443
      protocol: HTTPS
    tls:
      credentialName: httpbin-ssl-certificate-secret
      mode: SIMPLE

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  annotations:
    meta.helm.sh/release-name: httpbin-ingress
    meta.helm.sh/release-namespace: httpbin
  creationTimestamp: "2022-10-13T08:07:33Z"
  generation: 1
  labels:
    app.kubernetes.io/managed-by: Helm
  name: httpbin-ingress
  namespace: istio-ingress
  resourceVersion: "5246"
  uid: ef5b6397-2c7a-408c-b142-4528e8f28a20
spec:
  gateways:
  - httpbin-ingress
  hosts:
  - httpbin.example.com
  http:
  - match:
    - uri:
        prefix: /outpost.goauthentik.io
    route:
    - destination:
        host: authentik.authentik.svc.cluster.local
        port:
          number: 80
  - match:
    - uri:
        regex: ^\/[^\.]+.*
    - uri:
        exact: /
    route:
    - destination:
        host: httpbin.httpbin.svc.cluster.local
        port:
          number: 14001

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  annotations:
    kubernetes.io/ingress.class: istio
    nginx.ingress.kubernetes.io/whitelist-source-range: 0.0.0.0/0,::/0
  creationTimestamp: "2022-10-13T08:07:38Z"
  generateName: cm-acme-http-solver-
  generation: 1
  labels:
    acme.cert-manager.io/http-domain: "1703151793"
    acme.cert-manager.io/http-token: "1233129203"
    acme.cert-manager.io/http01-solver: "true"
  name: cm-acme-http-solver-gtgxg
  namespace: istio-ingress
  ownerReferences:
  - apiVersion: acme.cert-manager.io/v1
    blockOwnerDeletion: true
    controller: true
    kind: Challenge
    name: httpbin-ssl-certificate-ct48l-1136457683-1300359052
    uid: dd19a50c-5944-46b8-ae09-8345bef9c114
  resourceVersion: "5308"
  uid: 5d5578a5-3371-4705-9a8c-e031be5f4d7c
spec:
  rules:
  - host: httpbin.example.com
    http:
      paths:
      - backend:
          service:
            name: cm-acme-http-solver-rkr2g
            port:
              number: 8089
        path: /.well-known/acme-challenge/YKCZwQz6T9HezJtPwzev-esq-Q4WaLHoUC_CafmPJUk
        pathType: ImplementationSpecific
status:
  loadBalancer: {}

The problem I face is the following. With this setup:

  • curl --resolve httpbin.example.com:443:127.0.0.1 https://httpbin.example.com/ -k works.
  • curl --resolve httpbin.example.com:443:127.0.0.1 https://httpbin.example.com/.well-known/acme-challenge/YKCZwQz6T9HezJtPwzev-esq-Q4WaLHoUC_CafmPJUk -Ik gives http code 404.
  • if I delete the gateway httpbin-ingress, curl --resolve httpbin.example.com:80:127.0.0.1 http://httpbin.example.com/.well-known/acme-challenge/YKCZwQz6T9HezJtPwzev-esq-Q4WaLHoUC_CafmPJUk -Ik works as expected with http code 200.

The Certificate resource for cert-manager is annotated with

    cert-manager.io/issue-temporary-certificate: "true"

and that works (the gateway is setup with a self-signed certificate until letsencrypt succeeds), so the fact that I use httpsRedirect: true should not be the culprit.

My question is: is it possible to have the gateway in place and have cert-manager succeed with the HTTP01 challenge? My thinking is that there must be something I am overlooking in getting the gateway forward traffic for "/.well-known/..." to the cert-manager's ingress.

I looked at this question, Using Gateway + VirtualService + http01 + SDS , but I have not been able to find where my configuration is different. I tried changing the gateway's protocol on port 80 from HTTP to HTTP2, and curling with --http1.1 to the .well-known path, but this did not solve the issue.

wessel
  • 534
  • 5
  • 15

1 Answers1

0

The solution for me was in the end:

  • to add a * host to the servers on port 80 in each Gateway,
  • drop the use of the built-in httpsRedirect: true function,
  • and write a slightly more manual redirect rule for http to https which only matches the non-ACME paths.

It seems that you cannot have multiple Istio Gateways for the same port and hostname, unless the hostnames include *. Someone correct me if I'm wrong.

My configuration which works is now:

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  annotations:
    meta.helm.sh/release-name: httpbin-ingress
    meta.helm.sh/release-namespace: httpbin
  creationTimestamp: "2022-10-13T13:24:27Z"
  generation: 1
  labels:
    app.kubernetes.io/managed-by: Helm
  name: httpbin-ingress
  namespace: istio-ingress
  resourceVersion: "54782"
  uid: d36977db-20a2-4d43-a137-ba4cbfeccf8d
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - '*'
    - httpbin.example.com
    port:
      name: http
      number: 80
      protocol: HTTP
  - hosts:
    - httpbin.example.com
    port:
      name: https
      number: 443
      protocol: HTTPS
    tls:
      credentialName: httpbin-ssl-certificate-secret
      mode: SIMPLE

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  annotations:
    meta.helm.sh/release-name: httpbin-ingress
    meta.helm.sh/release-namespace: httpbin
  creationTimestamp: "2022-10-13T13:24:27Z"
  generation: 1
  labels:
    app.kubernetes.io/managed-by: Helm
  name: httpbin-ingress
  namespace: istio-ingress
  resourceVersion: "54783"
  uid: 3a1d988c-c287-49a8-942a-9aaf41a4b2b5
spec:
  gateways:
  - httpbin-ingress
  hosts:
  - httpbin.example.com
  http:
  - match:
    - headers:
        x-forwarded-proto:
          exact: https
      uri:
        prefix: /outpost.goauthentik.io
    route:
    - destination:
        host: authentik.authentik.svc.cluster.local
        port:
          number: 80
  - match:
    - headers:
        x-forwarded-proto:
          exact: https
      uri:
        regex: ^\/[^\.]+.*
    - headers:
        x-forwarded-proto:
          exact: https
      uri:
        exact: /
    route:
    - destination:
        host: httpbin.httpbin.svc.cluster.local
        port:
          number: 14001
  - match:
    - headers:
        x-forwarded-proto:
          exact: http
      uri:
        regex: ^\/[^\.]+.*
    - headers:
        x-forwarded-proto:
          exact: http
      uri:
        exact: /
    - headers:
        x-forwarded-proto:
          exact: http
      uri:
        exact: /
    redirect:
      scheme: https

And no need to repost the generated ingress from cert-manager, as that did not change!

As you can see, I elaborated the VirtualService quite a bit, where each match now explicitly verifies the protocol (http vs https), and the lines which match http only apply to paths which do not match the ACME Challenge (/.well-known...).

If your service uses paths that start with a dot, well, then you will have to add more rules to the matching to avoid matching ACME but allow for your .-paths.

PS If someone knows a smarter RE2-regex such that it fits in one line, please do tell!

wessel
  • 534
  • 5
  • 15