0

Can you please kindly help. I have successfully deployed KNative cluster for our ML services, using official documentation. Everything is working good, expect HTTPS. I've followed https://knative.dev/docs/serving/using-a-tls-cert/, installed cert-manager helm, configured DNS issuer with AWS Route53.

Certificates have been created successfully, I've limited it to only 2 namespaces

> kubectl get configmap config-network --namespace knative-serving -o yaml
apiVersion: v1
data:
  auto-tls: Enabled
  autocreate-cluster-domain-claims: "false"
  certificate-class: cert-manager.certificate.networking.knative.dev
  default-external-scheme: https
  domain-template: '{{.Name}}.{{.Namespace}}.{{.Domain}}'
  enable-mesh-pod-addressability: "false"
  http-protocol: Redirected
  ingress-class: istio.ingress.networking.knative.dev
  mesh-compatibility-mode: auto
  namespace-wildcard-cert-selector: '{"matchExpressions": [{"key":"networking.knative.dev/disableWildcardCert",
    "operator": "NotIn", "values":["true"]}]}'
  rollout-duration: "0"
  tag-template: '{{.Tag}}-{{.Name}}'
kind: ConfigMap
metadata:
  labels:
    app.kubernetes.io/component: networking
    app.kubernetes.io/name: knative-serving
    app.kubernetes.io/version: 1.4.0
  name: config-network
  namespace: knative-serving
> k get certificates -A
NAMESPACE         NAME                                         READY   SECRET                                       AGE
default           default.knative.example.com               True    default.knative.example.com               23h
default           route-073010ed-d843-4cdb-8d70-f32873459f2e   True    route-073010ed-d843-4cdb-8d70-f32873459f2e   23h
default           route-b534802c-c89b-4618-a214-93d25bf5c408   False   route-b534802c-c89b-4618-a214-93d25bf5c408   23h
istio-system      istio-system.knative.example.com          True    istio-system.knative.example.com          23h
knative-serving   knative-serving.knative.example.com       True    knative-serving.knative.example.com       23h
kserve            kserve.knative.example.com                True    kserve.knative.example.com                23h
kserve            serving-cert                                 True    kserve-webhook-server-cert                   40d
kube-system       aws-load-balancer-serving-cert               True    aws-load-balancer-tls                        41d

I did sample InferenceService deployment

apiVersion: serving.kserve.io/v1beta1
kind: InferenceService
metadata:
  name: mlclass
spec:
  predictor:
    containers:
    - name: kserve-container
      image: ...
      resources:
        limits:
          cpu: "1"
          memory: 4Gi
          nvidia.com/gpu: "1"
        requests:
          cpu: "1"
          memory: 4Gi
          nvidia.com/gpu: "1"

All resources seem to be created, but it works only via http

curl -v --location 'http://mlclass.default.knative.example.com/v1/models/mlclass:predict'

Then i have tried to enable HTTPS. Namespace wildcard certificate has been issued, everything seem to be fine, but when i try to reach my service with https, it shows 404 for common url like https://mlclass.default.knative.example.com/v1/models/mlclass:predict.

In the same time, when i try to reach specific predictor URL, it works.

> k get ksvc
mlclass-predictor-default   https://mlclass-predictor-default.default.knative.example.com   mlclass-predictor-default-00001   mlclass-predictor-default-00001   True

I have followed this guide https://knative.dev/docs/serving/using-auto-tls/#enabling-auto-tls and changed gateway like this:

> k get gateways.networking.istio.io -A
NAMESPACE         NAME                                          AGE
default           mlclass-predictor-default-3797421420   2d20h
default           wildcard-3791198b                             3d19h
knative-serving   knative-ingress-gateway                       3d23h
knative-serving   knative-local-gateway                         4d23h
> k get gateways.networking.istio.io knative-ingress-gateway -n knative-serving -o yaml
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  annotations:
     ...
  labels:
    app.kubernetes.io/component: net-istio
    app.kubernetes.io/name: knative-serving
    app.kubernetes.io/version: 1.4.0
    networking.knative.dev/ingress-provider: istio
  name: knative-ingress-gateway
  namespace: knative-serving
spec:
  selector:
    istio: ingressgateway
  servers:
  - hosts:
    - '*'
    port:
      name: http
      number: 80
      protocol: HTTP
    tls:
      httpsRedirect: false
  - hosts:
    - '*'
    port:
      name: https
      number: 443
      protocol: HTTPS
    tls:
      mode: SIMPLE

I have added section for port 443. Can you please kindly point to the right direction, how to enable HTTPS for InferenceService in KNative?

dmitrii
  • 31
  • 1

1 Answers1

0

The main problem was pretty simple yet.

  • The wildcard certificate for namespaces has to be in the same namespace as the gateway (knative-ingress-gateway)
  • I have added additional HTTPS port configuration in the same knative-ingress-gateway since by default it has only http port)

So it's working as expected, i only had to perform these modifications after common installation guide.

dmitrii
  • 31
  • 1