Why is the port field in Istio Gateway important?

Question

I'm currently learning about Istio's Gateway, and there's something I don't quite understand.

So when we install istio, we get the Deployment istio-ingressgateway:

$ kubectl get deploy -n istio-system
NAME                   READY   UP-TO-DATE   AVAILABLE   AGE
istio-egressgateway    1/1     1            1           3h9m
istio-ingressgateway   1/1     1            1           3h9m
istiod                 1/1     1            1           3h9m

Now, this istio-ingressgateway is exposed via the Service:

$ kubectl get svc -n istio-system
NAME                   TYPE           CLUSTER-IP      EXTERNAL-IP                                                              PORT(S)                                                                      AGE
istio-egressgateway    ClusterIP      172.20.33.252   <none>                                                                   80/TCP,443/TCP                                                               3h10m
istio-ingressgateway   LoadBalancer   172.20.19.8     abc-xyz.us-east-1.elb.amazonaws.com   15021:30308/TCP,80:32116/TCP,443:30034/TCP,31400:30430/TCP,15443:30260/TCP   3h10m
istiod                 ClusterIP      172.20.27.14    <none>                                                                   15010/TCP,15012/TCP,443/TCP,15014/TCP                                        3h10m

Now, the proxy (istio-ingressgateway deployment's pods) listens on port 8080, which is the targetPort for port 80 in the istio-ingressgateway Service.

Now let's say I have the following Gateway configuration:

apiVersion: networking.istio.io/v1beta1
kind: Gateway
metadata:
  name: my-ingress
spec:
  selector:
    app: my-ingressgateway
  servers:
  - port:
      number: 80
      name: http2
      protocol: HTTP2
    hosts:
    - "www.example.com"

What I don't understand is, why is the server's port.number configuration is important, and on what it affects?

I'll explain: when I provision a EKS cluster for example and install istio on it, so I get a AWS Load Balancer for that istio-ingressgateway Service. One of the ports of the load balancer is port 80, which forwards the traffic to a nodePort, which is than routed to port 8080 in the proxy container:

...
    - name: http2
      nodePort: 32116
      port: 80
      protocol: TCP
      targetPort: 8080
...

So basically, when I access <load-balancer-domain>:80, I'm routed to one of the cluster nodes to port 32116, and from there to the appropriate pod, so the pod doesn't even know what port I as a client used to reach it.

My question is then, what does the Gateway configuration "care" about what port I use to reach the proxy container?

EDIT

I'll try to rephrase the question: The istio-ingressgateway pod listens on port 8080. The istio-ingressgateway Service routes traffic to this port. If so, what is the meaning of the .servers[].port configuration in the Ingress resource?

The documentation about Gateway says:

While Istio will configure the proxy to listen on these ports, it is the responsibility of the user to ensure that external traffic to these ports are allowed into the mesh.

And I ask: What do they mean by "Istio will configure the proxy to listen on these ports"? The proxy listens on the ports configured in the pod's specification, for example 8080 for HTTP, and the Service is the one that specifies which port will forward traffic to these ports in the pod, so what is the meaning of this port number in the Ingress resource ?!?!

Hope it's more clear now

score 0 · Answer 1 · answered May 04 '23 at 16:30

0

The configuration of Gateway (and also VirtualService and DestinationRule) are abstractions for envoy. They don't configure kubernetes but the envoys that run in the istio-ingressgateway (and pod sidecar) containers.

With your my-ingress gateway manifest you simple tell istio: Configure the istio-ingressgateway that runs in a pod matching the selector app: my-ingressgateway to listen to requests for host www.example.com on port 80.

You can check out the envoy config with the istioctl proxy-config command, like

istioctl proxy-config all my-ingress (add -o json for a more detailed output (but in json)). See also docs about that command.

So it's actually:

Request <load-balancer-domain>:80 -> <nodeIP>:32116 -> istio-ingressgateway pod -> envoy config determines how to handle the request for <load-balancer-domain>:80 (= Gateway + VirtualService for routing + DestinationRule, EnvoyFilter, etc) -> Forward to pod -> istio sidecar handles request, based on envoy config (EnvoyFilter, Sidecar, RequestAuthentication, etc) -> Forward to main application in the pod

If you have access to the kubernetes node you can run iptables -t nat -L KUBE-NODEPORTS | grep istio-ingressgateway to see and follow the actual routing.

answered May 04 '23 at 16:30

Chris

5,109
3
19
40

Hi @Chris. The thing I'm trying to understand is: You already specify the port of the `Service` which will bring you to the envoy that runs in the pod, so why do you need to configure the same port in the `Gateway` configuration? And, more specifically, how do the envoy "know" that the client did use that port, if we have this NATting on the way? – YoavKlein May 04 '23 at 18:39
Envoy isn't auto-configured based on the `Service`. That would be a bad idea, because how should it know what you want to do for the different port destinations? tls redirect on port 80 or not? tls termination or passthrough on port 433? Which protocol will be served at the port? What if there is a non standard port? That's why you need to configure the ingress-gateway, via the `Gateway` resource. Regarding NATing etc. Not sure about how ELB handles this, but the `host:port` destination reaches the envoy. I'd guess that the `HOST` header is preserved when forwarding the request to the cluster – Chris May 05 '23 at 05:38
yes, the `Host` header is preserved, my question is about the port. If you look at the envoy Deployment, you'll see that it listens on port 8080, and the Service forwards traffic that it receives on port 80 to 8080. So my question is, how do the Envoy "know" which port the client used to access it? The description "The Gateway configures the proxy to listen on.." is not understood, since the proxy "listens on", technically, port 8080, so the Gateway cannot configure it to listen on any other port. – YoavKlein May 05 '23 at 06:05
B.T.W, when trying to change the port of `istio-ingressgateway` to something else, say, 81, I was able to access the application just the same, although it did stop routing traffic after a while. However, if I changed the `port` configuration in the Gateway, it did stop routing traffic immediately.. – YoavKlein May 05 '23 at 06:07
I think I understood something now: The Envoy pod is configured to receive HTTP traffic on port 8080, and whatever it receives on this port is assumed to be "port 80", although it doesn't have to be. That's why when I change the port in the Gateway config to 81, it stops routing my traffic. But when I change the `http2` port in the `istio-ingressgateway` Service to 81 for example, traffic still arrives! (although it stop arriving after a while, which I still don't understand why). Do you think it's correct @Chris? – YoavKlein May 05 '23 at 06:19
Yes, I think you're confusing/mixing istio config with kubernetes config. I'd suggest debug istio config with `istioctl pc all -o json` before and after applying the gateway manifest to see what changes in the envoy config. – Chris May 05 '23 at 07:14

score 0 · Answer 2 · answered May 06 '23 at 21:05

Ok I think I understand how it works.

Let's set the stage: we have the istio-ingressgateway Deployment, which runs the Envoy proxy pod, which is our Gateway. This pod listens on a set of ports, such as 9443, 8080, and few other. Additionally, there is a Service that serves this Deployment, which exposes the above-mentioned ports by ports of the Service. For example, port 8080 of the pod is exposed by port 80 of the Service. Now, we create a Gateway resource which defines a server that listens on port 80.

Now ask yourself: The Envoy pod listens on port 8080, not 80. But the server configuration in Gateway specifies port 80. How does it know that traffic received on port 8080 should be handled as HTTP, as specified in the Gateway?

Apparently, Istio (I guess it's istiod) reads the Gateway configuration, and if it sees a server with a port that corresponds to a port in the Envoy's Service (80), than it configures the corresponding port of the Envoy (8080) to listen for the specified protocol and settings.

This explains the following behaviors:

If you change the port in the Gateway configuration, the application is unreachable immediately. This is because there's no matching between the port in the Service and the port in the Gateway, so istio says: ok port 8080 should not apply these server's settings anymore.
If you changee the port in the Service configuration, you will still reach the application for a while (about 10-20 minutes). After this it stops. This is apparently because istio polls the Service's configuration every once in a while, and doesn't get notified immediately like it does with the Gateway. If you now change the port in the Gateway to the modified port in the Service, you will get traffic.

Hope that's clear

Why is the port field in Istio Gateway important?

EDIT

2 Answers2