Istio mTLS make the service unavailable (503)

Question

I am trying to implement Istio mTLS migration in my kubernetes cluster. The problem is, when I create the "PeerAuthentication" manifest with the option "STRICT" to enforce encrypted traffic it makes my backend service unavailable (503). With PERMISSIVE mode, everything seems to be working fine.

Basic rundown of my traffic flow: Outside Proxy (hosted on a different cloud) -> Network Load Balancer -> Gateway (Istio) -> VirtualService -> FrontEnd Service -> Backend (API Calls) -> Postgres

PeerAuthentication.yaml

apiVersion: security.istio.io/v1beta1
kind: PeerAuthentication
metadata:
  name: mtls
  namespace: dev
spec:
  mtls:
    mode: STRICT

Gateway.yaml

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: gateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
    - port:
        number: 80
        name: http
        protocol: HTTP
      hosts:
        - "*"
      tls:
        httpsRedirect: true
    - port:
        number: 443
        name: https-443
        protocol: HTTPS
      hosts:
        - "*"
      tls:
        mode: MUTUAL
        serverCertificate: /etc/istio/ingressgateway-certs/tls.crt
        privateKey: /etc/istio/ingressgateway-certs/tls.key
        caCertificates: /etc/istio/ingressgateway-ca-certs/proxy-certs.crt

VirtualService.yaml

apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: frontend-dev
  namespace: dev
spec:
  gateways:
  - istio-system/gateway
  hosts:
  - "url-to-my-website"
  http:
  - route:
    - destination:
        host: frontend-dev.dev.svc.cluster.local

In short terms, I want that my traffic would be encrypted inside the cluster, since on the outside I am already using TLS for accessing the website.

From the proxy logs, I see that the first contact does make through, since the FE is returning 503 with the call to backend:

[2023-03-13T10:01:06.801Z] "POST /ac-be/api/token HTTP/1.1" 503 - via_upstream - "-"

And the BE returns a "Filter Chain Not Found":

[2023-03-07T16:30:57.589Z] "- - -" 0 NR filter_chain_not_found - "-" 0 0 0 - "-" "-" "-" "-" "-" - - 172.30.21.37:8080 172.30.21.22:60372 - -

Additionally I tried setting up a Destination Rule for both FE and BE, but that didn't help

DestinationRuleFe.yaml

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: fe-mtls
  namespace: dev
spec:
  host: "frontend-dev.dev.svc.cluster.local"
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL

DestinationRuleBe.yaml

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: be-mtls
  namespace: dev
spec:
  host: "backend-dev.dev.svc.cluster.local"
  trafficPolicy:
    tls:
      mode: ISTIO_MUTUAL