I am new to istio and i am trying to enable the STRICT mode of mTLS at the namespace level i.e
apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
name: prod
namespace: prod
spec:
mtls:
mode: STRICT
Everything seems to be working fine when the mode was set to PERMISSIVE.
Let me give you a flow of my api.
istio-gateway(tls terminated) -> virtual-service-> myservice-gateway(spring cloud-gateway svc1) -> another-service-svc2 (through load balancing). So when i hit some endpoint in the myservice-gateway(svc1), it returns succesfully but when it calls other services i get upstream connect error or disconnect/reset before headers. reset reason: connection termination. This happens when i call other services from one service.
When i check the logs of my svc2 istio-proxy i get:
[2022-10-31T00:10:46.935Z] "- - -" 0 NR filter_chain_not_found - "-" 0 0 0 - "-" "-" "-" "-" "-" - - 172.17.0.10:8084 172.17.0.8:46738 - -
[2022-10-31T00:10:46.966Z] "- - -" 0 NR filter_chain_not_found - "-" 0 0 0 - "-" "-" "-" "-" "-" - - 172.17.0.10:8084 172.17.0.8:46742 - -
[2022-10-31T00:10:47.001Z] "- - -" 0 NR filter_chain_not_found - "-" 0 0 0 - "-" "-" "-" "-" "-" - - 172.17.0.10:8084 172.17.0.8:46746 - -
[2022-10-31T00:10:51.746Z] "- - -" 0 NR filter_chain_not_found - "-" 0 0 0 - "-" "-" "-" "-" "-" - - 172.17.0.10:8084 172.17.0.8:46864 - -
[2022-10-31T00:10:51.782Z] "- - -" 0 NR filter_chain_not_found - "-" 0 0 0 - "-" "-" "-" "-" "-" - - 172.17.0.10:8084 172.17.0.8:46868 - -
[2022-10-31T00:10:51.846Z] "- - -" 0 NR filter_chain_not_found - "-" 0 0 0 - "-" "-" "-" "-" "-" - - 172.17.0.10:8084 172.17.0.8:46872 - -
[2022-10-31T00:10:52.048Z] "- - -" 0 NR filter_chain_not_found - "-" 0 0 0 - "-" "-" "-" "-" "-" - - 172.17.0.10:8084 172.17.0.8:46876 - -
[2022-10-31T00:10:52.067Z] "- - -" 0 NR filter_chain_not_found - "-" 0 0 0 - "-" "-" "-" "-" "-" - - 172.17.0.10:8084 172.17.0.8:46882 - -
[2022-10-31T00:10:52.113Z] "- - -" 0 NR filter_chain_not_found - "-" 0 0 0 - "-" "-" "-" "-" "-" - - 172.17.0.10:8084 172.17.0.8:46888 - -
[2022-10-31T00:10:52.308Z] "- - -" 0 NR filter_chain_not_found - "-" 0 0 0 - "-" "-" "-" "-" "-" - - 172.17.0.10:8084 172.17.0.8:46898 - -
The ingress gateway log shows:
[2022-10-31T00:13:08.800Z] "GET /transaction/actuator HTTP/2" 503 URX via_upstream - "-" 0 95 110 108 "172.17.0.1" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/100.0.4896.75 Safari/537.36 Edg/100.0.1185.36" "90b95e7a-2a34-94f3-b804-a1922d529d82" "api-gateway.com" "172.17.0.8:8765" outbound|8765||api-gateway.prod.svc.cluster.local 172.17.0.22:51514 172.17.0.22:8443 172.17.0.1:31955 api-gateway.com -
Below are my kubernetes file:
apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
name: mygateway
namespace: prod
spec:
selector:
istio: ingressgateway
servers:
- hosts:
- '*'
port:
name: https
number: 443
protocol: HTTPS
tls:
credentialName: secret-tls
mode: SIMPLE
apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
name: api-gateway
namespace: prod
spec:
gateways:
- mygateway
hosts:
- api-gateway.com
http:
- match:
- uri:
prefix: /
route:
- destination:
host: api-gateway.prod.svc.cluster.local
port:
number: 8765
apiVersion: networking.istio.io/v1beta1
kind: DestinationRule
metadata:
name: rule
namespace: prod
spec:
host: '*.local'
trafficPolicy:
tls:
mode: ISTIO_MUTUAL
my svc1 file
apiVersion: apps/v1
kind: Deployment
metadata:
name: api-gateway
namespace: prod
labels:
app: api-gateway
version: v1
spec:
replicas: 1
selector:
matchLabels:
app: api-gateway
version: v1
template:
metadata:
labels:
app: api-gateway
version: v1
spec:
serviceAccountName: api-gateway
containers:
- name: api-gateway
image: sunday/api-gateway
imagePullPolicy: Never
ports:
- containerPort: 8765
livenessProbe:
httpGet:
path: /actuator/health/liveness
port: 8765
initialDelaySeconds: 300
periodSeconds: 5
readinessProbe:
httpGet:
path: /actuator/health/readiness
port: 8765
initialDelaySeconds: 300
periodSeconds: 10
---
apiVersion: v1
kind: Service
metadata:
labels:
app: api-gateway
service: api-gateway
name: api-gateway
namespace: prod
spec:
ports:
- name: http
port: 8765
targetPort: 8765
selector:
app: api-gateway
---
apiVersion: v1
kind: ServiceAccount
metadata:
name: api-gateway
namespace: prod
labels:
account: api-gateway
my svc 2
apiVersion: apps/v1
kind: Deployment
metadata:
name: transaction
namespace: prod
labels:
app: transaction
version: v1
spec:
replicas: 1
selector:
matchLabels:
app: transaction
version: v1
template:
metadata:
labels:
app: transaction
version: v1
spec:
serviceAccountName: transaction
containers:
- name: transaction
image: sunday/transaction
ports:
- containerPort: 8084
livenessProbe:
httpGet:
path: /actuator/health/liveness
port: 8084
initialDelaySeconds: 300
periodSeconds: 5
readinessProbe:
httpGet:
path: /actuator/health/readiness
port: 8084
---
apiVersion: v1
kind: Service
metadata:
labels:
app: transaction
service: transaction
name: transaction
namespace: prod
spec:
ports:
- name: http
port: 8084
targetPort: 8084
selector:
app: transaction
---
apiVersion: v1
kind: ServiceAccount
metadata:
namespace: prod
name: transaction
labels:
account: transaction
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: transaction
namespace: prod
spec:
hosts:
- transaction.prod.svc.cluster.local
gateways:
- mesh
http:
- match:
- uri:
prefix: /
route:
- destination:
port:
number: 8084
host: transaction.prod.svc.cluster.local
istioctl proxy-config cluster -n istio-system istio-ingressgateway-66d5c58595-7n9r5
SERVICE FQDN PORT SUBSET DIRECTION TYPE DESTINATION RULE
BlackHoleCluster - - - STATIC
agent - - - STATIC
api-gateway.prod.svc.cluster.local 8765 - outbound EDS transaction.prod
cloud-config.prod.svc.cluster.local 8888 - outbound EDS transaction.prod
customer.prod.svc.cluster.local 8081 - outbound EDS customer.prod
dashboard-metrics-scraper.kubernetes-dashboard.svc.cluster.local 8000 - outbound EDS
docker-mysql.prod.svc.cluster.local 3306 - outbound EDS transaction.prod
employee.prod.svc.cluster.local 8082 - outbound EDS transaction.prod
eureka.prod.svc.cluster.local 8761 - outbound EDS transaction.prod
github.com 443 - outbound STRICT_DNS transaction.prod
ingress-nginx-controller-admission.ingress-nginx.svc.cluster.local 443 - outbound EDS
ingress-nginx-controller.ingress-nginx.svc.cluster.local 80 - outbound EDS
ingress-nginx-controller.ingress-nginx.svc.cluster.local 443 - outbound EDS
istio-egressgateway.istio-system.svc.cluster.local 80 - outbound EDS
istio-egressgateway.istio-system.svc.cluster.local 443 - outbound EDS
istio-ingressgateway.istio-system.svc.cluster.local 80 - outbound EDS
istio-ingressgateway.istio-system.svc.cluster.local 443 - outbound EDS
istio-ingressgateway.istio-system.svc.cluster.local 3306 - outbound EDS
istio-ingressgateway.istio-system.svc.cluster.local 8765 - outbound EDS
istio-ingressgateway.istio-system.svc.cluster.local 15021 - outbound EDS
istio-ingressgateway.istio-system.svc.cluster.local 15443 - outbound EDS
istio-ingressgateway.istio-system.svc.cluster.local 31400 - outbound EDS
istio-operator.istio-operator.svc.cluster.local 8383 - outbound EDS
istiod.istio-system.svc.cluster.local 443 - outbound EDS
istiod.istio-system.svc.cluster.local 15010 - outbound EDS
istiod.istio-system.svc.cluster.local 15012 - outbound EDS
istiod.istio-system.svc.cluster.local 15014 - outbound EDS
keycloak-headless.prod.svc.cluster.local 443 - outbound EDS transaction.prod
keycloak-headless.prod.svc.cluster.local 8080 - outbound EDS transaction.prod
keycloak.prod.svc.cluster.local 443 - outbound EDS transaction.prod
keycloak.prod.svc.cluster.local 8080 - outbound EDS transaction.prod
kiali.istio-system.svc.cluster.local 9090 - outbound EDS
kiali.istio-system.svc.cluster.local 20001 - outbound EDS
kube-dns.kube-system.svc.cluster.local 53 - outbound EDS
kube-dns.kube-system.svc.cluster.local 9153 - outbound EDS
kubernetes-dashboard.kubernetes-dashboard.svc.cluster.local 80 - outbound EDS
kubernetes.default.svc.cluster.local 443 - outbound EDS
loan.prod.svc.cluster.local 8083 - outbound EDS transaction.prod
metrics-server.kube-system.svc.cluster.local 443 - outbound EDS
prometheus.istio-system.svc.cluster.local 9090 - outbound EDS
prometheus_stats - - - STATIC
rabbitmq.rabbitmq.svc.cluster.local 5672 - outbound EDS
rabbitmq.rabbitmq.svc.cluster.local 15672 - outbound EDS
sds-grpc - - - STATIC
tracing.istio-system.svc.cluster.local 80 - outbound EDS
transaction.prod.svc.cluster.local 8084 - outbound EDS transaction.prod
xds-grpc - - - STATIC
zipkin - - - STRICT_DNS
zipkin.istio-system.svc.cluster.local
My proxy-config listeners for my api-gateway is here
Please i need all the help i can get.
