Apply envoyfilter ext_authz per virtual hosts

Question

I have ext_authz filter as this:

kind: EnvoyFilter
metadata:
  name: authn-filter
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      istio: ingressgateway
  configPatches:
  - applyTo: HTTP_FILTER
    match:
      context: GATEWAY
      listener:
        filterChain:
          filter:
            name: "envoy.http_connection_manager"
            subFilter:
              name: "envoy.router"
    patch:
      operation: INSERT_BEFORE
      value:
        name: envoy.ext_authz
        config:
          http_service:
            server_uri:
              uri: http://authservice.istio-system.svc.cluster.local
              cluster: outbound|8080||authservice.istio-system.svc.cluster.local
              failure_mode_allow: false
              timeout: 10s
            authorization_request:
              allowed_headers:
                patterns:
                - exact: "cookie"
                - exact: "X-Auth-Token"
            authorization_response:
              allowed_upstream_headers:
                patterns:
                - exact: "kubeflow-userid"
          status_on_error:
            code: GatewayTimeout

The problem is that it applies to all virtual hosts under the same ingress. I want to only apply it to specific virtual hosts. Currently I'm excluding some hosts using the following:

apiVersion: networking.istio.io/v1alpha3
kind: EnvoyFilter
metadata:
  name: bypass-auth-filter
  namespace: istio-system
spec:
  workloadSelector:
    labels:
      istio: ingressgateway
  configPatches:
  - applyTo: VIRTUAL_HOST
    match:
      routeConfiguration:
        vhost:
          name: a.example.com:80
    patch:
        operation: MERGE
        value:
          per_filter_config:
            envoy.ext_authz:
              disabled: true

However, I would like if I can do it such that I apply the filter on specific virtual hosts rather than having to exclude every hosts that doesn't need auth (more like a whitelist solution rather than a blacklist)

Any hints would be greatly appreciated!