1

Having trouble with istio ingress gateway. We configured gateway in SIMPLE tls mode for traffic coming into the mesh. But we also want istio ingress gateway to talk to application services(with istio-proxy sidecar) with mTLS. Virtual Services are configured such that requests from whitelisted hosts is navigated to services in processing namespace. istio ingress pods are in istio-system (no istio-proxy sidecar). gateway and istio ingress gateway pods are also in istio-system. We want to apply RBAC on processing namespace workloads as follows-

  rules:
  - from:
    - source:
        principals: [
          "cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account"
        ]

Peer Authentication Policy on processing as well as istio-system is mTLS in permissive mode. Destination rule in istio-system-

apiVersion: "networking.istio.io/v1alpha3"
kind: "DestinationRule"
metadata:
  name: "istio-mutual"
  namespace: "istio-system"
spec:
  host: "*.svc.cluster.local"
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL

but this is not working as expected. Logs from istio-proxy container of workload pod when curl-ed from istio ingress gateway pod to workload in processing namespace-

curl myservice.processing/healthz
    2021-01-28T23:29:47.105404Z debug   envoy rbac  checking connection: requestedServerName: , sourceIP: 10.107.113.175:49912, directRemoteIP: 10.107.113.175:49912,remoteIP: 10.107.113.175:49912, localAddress: 10.107.66.26:8080, ssl: none, dynamicMetadata: 
    [2021-01-28T23:29:47.105Z] "- - -" 0 - "-" "-" 92 0 0 - "-" "-" "-" "-" "127.0.0.1:8080" inbound|80|https-web|myservice.processing.svc.cluster.local 127.0.0.1:39048 10.107.66.26:808010.107.113.175:49912 - -

How should we configure SIMPLE TLS for downstream and ISTIO_MUTUAL for connecting with upstream on gateway object? istio ingress gateway also has mTLS identity. but in the log, I see ssl: None. I dont really understand why. Has anybody faced this kind of issue?

EDITED curl command output

curl -v myservice.processing/actuator/info
*   Trying 172.20.252.59...
* TCP_NODELAY set
* Connected to myservice.processing (172.20.252.59) port 80 (#0)
> GET /actuator/info HTTP/1.1
> Host: myservice.processing
> User-Agent: curl/7.58.0
> Accept: */*
> 
* Empty reply from server
* Connection #0 to host myservice.processing left intact
curl: (52) Empty reply from server

Authentication Policy

apiVersion: authentication.istio.io/v1alpha1
  kind: Policy
  metadata: 
    name: default
    namespace: processing
  spec: 
    peers: 
    - mtls: {}

Gateway.yaml

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: istio-gateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 443
      name: https-port
      protocol: HTTPS
    hosts:
    - "*"
    tls:
      mode: SIMPLE
      minProtocolVersion: TLSV1_2
      credentialName: ingress-cert

virtual service.yaml

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata: 
  name: rate-limiting-gate
  namespace: processing
spec: 
  gateways: 
  - istio-system/istio-gateway
  hosts: 
  - *
  http: 
  - match: 
    - port: 443
    route: 
    - destination: 
        host: rate-limiting-gate
        port: 
          number: 80

pact.yaml

apiVersion: "security.istio.io/v1beta1"
kind: "AuthorizationPolicy"
metadata:
  name: "{{ .Values.service.name }}-authorization-policy"
  namespace: {{ .Release.Namespace | quote }}
spec:
  selector:
    matchLabels:
      service: {{ .Values.service.name }}
  rules:
  - from:
    - source:
        principals: [
        "cluster.local/ns/istio-system/sa/istio-ingressgateway-service-account"
        ]
  - to:
    - operation:
        ports:
        - "9292"
Shadja Chaudhari
  • 253
  • 1
  • 3
  • 6
  • 1
    1.Does the curl command work? If it doesn't work what is the error code from `curl -v`? 2.Could you [verify](https://preliminary.istio.io/latest/docs/tasks/security/authentication/authn-policy/#auto-mutual-tls) if there is `X-Forwarded-Client-Cert` header used? That header’s presence is evidence that mutual TLS is used. You use `STRICT` or `PERMISSIVE` [mtls](https://istio.io/latest/docs/tasks/security/authentication/mtls-migration/)? – Jakub Jan 29 '21 at 12:02
  • Curl command does not work. I edited the question and have now included the output and also the authentication policy. – Shadja Chaudhari Jan 30 '21 at 18:14
  • 1
    Which version of Istio? Also, when the source and tartget both are running istio-proxy then istio by default does an mTLS. Are you saying that the Ingress Gateway POD making connection to your actual service (with istio sidecar) is not mTLS? – nitgeek Feb 01 '21 at 06:07
  • `curl: (52) Empty reply from server` means that curl is asked to do plain HTTP on a server that does HTTPS, could you please check if it works correctly with any http backend? Could you also add your istio gateway and virtual service yamls? – Jakub Feb 02 '21 at 07:10
  • @nitgeek you are right. in this case, source is istio-gateway and destination is pod with istio-proxy sidecar. hence no mTLS in the picture. – Shadja Chaudhari Mar 12 '21 at 09:39
  • @Jakub that same curl command works when I remove AuthorizationPolicy. It also works for one of the http backend(pact) where we have whitelisted port. Refer to pact.yaml. – Shadja Chaudhari Mar 12 '21 at 09:39
  • I found this issue on github which closely explains what we are facing https://github.com/istio/istio/issues/19975. – Shadja Chaudhari Mar 12 '21 at 09:41

0 Answers0