2

Kubernetes has a support of Pod load-balancing, session affinity through its kube-proxy. Kubernetes’ kube-proxy is essentially an L4 load balancer so we cannot rely on it to load balance L7-transport, e.g. muliple gRPC live connections or load-balancing based on http-headers, cookies, etc.

Service Mesh implementation like e.g. istio can handle these patterns on L7-level including gRPC. But I always thought that Service Mesh is just another layer on top of Kubernetes with additional capabilities(encrypted traffic, blue/green deployments/etc). E.g. My assumption always was that Kubernetes applications should be able to work on both vanilla Kubernetes without Mesh (e.g. for development/testing) or with a Mesh on. Adding this advanced traffic management on L7 breaks this assumption. I won't be able to work on a vanilla Kubernetes anymore, I will be tied to a specific implementation of Istio dataplane(Envoy).

Please let know if my assumption is correct or why not? There's not much information about this type of separation of concerns on this internet.

Ivan Voroshilin
  • 5,233
  • 3
  • 32
  • 61
  • i think ideally yes, application should be able to work on both vanilla k8s and without mesh. however, after some point, it might get hard to manage requirements if you have any type of circuit breaker, blue-green deployment, mTLS, canary deployment etc. – Harsh Manvar Feb 21 '22 at 18:51
  • Which version of Kubernetes and Istio did you use and how did you set up the cluster? Did you use bare metal installation or some cloud provider? It is important to reproduce your problem. – Mikołaj Głodziak Feb 22 '22 at 14:03
  • You can take the latest k8s/Istio if this helps or other mesh which handles L7 traffic. It doesn't matter much if they are in the cloud on on-prem because kube-proxy has always worked on L4. The question is about an architectural misalignment of kube-proxy(L4) and L7 traffic management in a service Mesh. – Ivan Voroshilin Feb 22 '22 at 16:43

1 Answers1

1

Let me refer first to the following statement of yours:

My assumption always was that Kubernetes applications should be able to work on both vanilla Kubernetes without Mesh (e.g. for development/testing) or with a Mesh on. Adding this advanced traffic management on L7 breaks this assumption.

I have different view at that, Service Meshes are transparent to the application, so they don't break anything in them, but just add an extra (network, security, monitoring) functions at no cost (ok, the cost it quite complex configuration from Mesh operator perspective). The Service Mesh like Istio doesn't need to occupy all K8S namespaces, so you can still have mixed type of workloads in your cluster (with and w/o proxies). If we speak about Istio, to enable full interoperatbility between them (mixed workloads) you may combine two its features together:

  • Peer authentication set to PERMISSIVE, so that workloads without sidecars (proxy) can accept both mutual TLS and plain text traffic.
  • Manual protocol selection, e.g. if you prefer your app speak raw TCP instead of app protocol determined by Envoy itself (e.g. http) - to avoid extra decorations injected by proxy to intercepted requests.

Alternatively you can write your own custom tcp_proxy EnvoyFilter to use Envoy as a L4 network proxy.

References:

https://istio.io/latest/docs/reference/config/networking/envoy-filter/ https://istio.io/latest/docs/concepts/security/#peer-authentication https://istio.io/latest/docs/ops/configuration/traffic-management/protocol-selection/

Nepomucen
  • 4,449
  • 3
  • 9
  • 24