Unable to get raw TCP traffic through Istio Gateway

Question

I can route HTTP traffic (e.g. Elasticsearch and various dashboards) through Istio Gateway, but I can't get raw TCP traffic through. I have two examples below (postgres and redpanda). I have no trouble accessing the underlying services (mypostgres.default.svc.cluster.local and three-node-cluster-0.three-node-cluster.redpanda-system.svc.cluster.local) internally with postgres and kafka clients.

My Gateway:

apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: gateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:
  - port:
      number: 80
      name: http
      protocol: HTTP
    hosts:
    - 'mydomain.cloud'
    tls:
      httpsRedirect: true
  - port:
      number: 443
      name: https
      protocol: HTTPS
    hosts:
      - 'mydomain.cloud'
    tls:
      mode: SIMPLE
      credentialName: letsencrypt-staging-tls
  - port:
      number: 9092
      name: redpanda
      protocol: TCP
    hosts:
    - 'mydomain.cloud'
  - port:
      number: 5432
      name: postgres
      protocol: TCP
    hosts:
    - 'mydomain.cloud'

Postgres spec:

apiVersion: kubegres.reactive-tech.io/v1
kind: Kubegres
metadata:
  name: mypostgres
  namespace: postgres

spec:

   replicas: 3
   image: postgres:13.2

   database:
      size: 50Gi
      storageClassName: postgres

   env:
      - name: POSTGRES_PASSWORD
        valueFrom:
           secretKeyRef:
              name: postgressecret
              key: superUserPassword

      - name: POSTGRES_REPLICATION_PASSWORD
        valueFrom:
           secretKeyRef:
              name: postgressecret
              key: replicationUserPassword

Virtual service:

spec:
  hosts:
  - "*"
  gateways:
  - istio-system/gateway
  tcp:
  - match:
    - port: 5432
    route:
    - destination:
        host: mypostgres.default.svc.cluster.local
        port:
          number: 5432

Destination rule

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: postgres-destination-rule
  namespace: default
spec:
  host: mypostgres.default.svc.cluster.local
  trafficPolicy:
    tls:
      mode: DISABLE

Redpanda

apiVersion: redpanda.vectorized.io/v1alpha1
kind: Cluster
metadata:
  name: three-node-cluster
spec:
  image: "vectorized/redpanda"
  version: "latest"
  replicas: 2
  resources:
    requests:
      cpu: 1
      memory: 2Gi
    limits:
      cpu: 1
      memory: 2Gi
  configuration:
    rpcServer:
      port: 33145
    kafkaApi:
    - port: 9092
    pandaproxyApi:
    - port: 8082
    adminApi:
    - port: 9644
    developerMode: true
  storage:
    storageClassName: redpanda

Virtual service

apiVersion: networking.istio.io/v1beta1
kind: VirtualService
metadata:
  name: redpanda-vts
  namespace: redpanda-system
spec:
  hosts:
  - "*"
  gateways:
  - istio-system/gateway
  tcp:
  - match:
    - port: 9092
    route:
    - destination:
        host: three-node-cluster-0.three-node-cluster.redpanda-system.svc.cluster.local
        port:
          number: 9092

Destination rule:

apiVersion: networking.istio.io/v1alpha3
kind: DestinationRule
metadata:
  name: redpanda-destination-rule
  namespace: redpanda-system
spec:
  host: three-node-cluster-0.three-node-cluster.redpanda-system.svc.cluster.local
  trafficPolicy:
    tls:
      mode: DISABLE

Any ideas? I've tried playing around with the host names, using asterisks instead of domain names, but no effect. Getting TLS will be another day's fight, but now I'd just like to get some traffic through.

For example, the following works for RedPanda from inside the cluster with the standard kafka-python client:

from kafka.admin import KafkaAdminClient, NewTopic

nodes = {'bootstrap.servers':'three-node-cluster-0.three-node-cluster.redpanda-system.svc.cluster.local, three-node-cluster-1.three-node-cluster.redpanda-system.svc.cluster.local'}

admin_client = KafkaAdminClient(
    bootstrap_servers=nodes['bootstrap.servers'],
    client_id='test'
)

topic_list = []
topic_list.append(NewTopic(name="test-topic", num_partitions=1, replication_factor=1))
admin_client.create_topics(new_topics=topic_list, validate_only=False)

Similarly, I would like to be able to do the following from outside K8s through Istio Gateway:

from kafka.admin import KafkaAdminClient, NewTopic

nodes = {'bootstrap.servers':'mydomain.cloud/kafka'}

admin_client = KafkaAdminClient(
    bootstrap_servers=nodes['bootstrap.servers'],
    client_id='test'
)

topic_list = []
topic_list.append(NewTopic(name="test-topic", num_partitions=1, replication_factor=1))
admin_client.create_topics(new_topics=topic_list, validate_only=False)

Answer 1

Based on the documentation about Istio Protocol Selection

Istio supports proxying any TCP traffic. This includes HTTP, HTTPS, gRPC, as well as raw TCP protocols. In order to provide additional capabilities, such as routing and rich metrics, the protocol must be determined. This can be done automatically or explicitly specified.

And the answer to your problem should be in this fragment:

Protocols can be specified manually in the Service definition.

This can be configured in two ways:

• By the name of the port: name: <protocol>[-<suffix>].
• In Kubernetes 1.18+, by the appProtocol field: appProtocol: <protocol>.

Note that behavior at the Gateway differs in some cases as the gateway can terminate TLS and the protocol can be negotiated.

Look at the example yaml:

Below is an example of a Service that defines a https port by appProtocol and an http port by name:

kind: Service
metadata:
  name: myservice
spec:
  ports:
  - number: 3306
    name: database
    appProtocol: https <-change here 'https' to 'tcp'
  - number: 80
    name: http-web

In your situation try to replace appProtocol: https and put appProtocol: tcp in your Service yaml

Bare in mind that Server First protocols, such as MySQL, are incompatible with automatic protocol selection. See Server first protocols for more information.