1

I've been looking at setting up an Ingress controller for a bare-metal Kubernetes cluster. I started looking at Ingress controllers, but these seem to only work well for HTTP services that are reachable via port 80 or 443. If you need to expose a TCP or UDP service on an arbitrary port, it seems possible to do with the Nginx or HAProxy Ingress controllers, but your cluster ends up sharing a single port range. Please let me know if I've misunderstood this.

If you need to expose and load balance TCP or UDP services on arbitrary ports, how would you do it? I was thinking of using ClientIP so that services get their own VIP and can use any ports they want, but the question then becomes, how do you route traffic to those VIPs and give them friendly DNS names? Is there a solution for this already, or do you have to build one yourself? Using NodePort or any solution that means namespaces have to share a single port range isn't really scalable or desirable. Especially if Bob in namespace 1 absolutely needs his service to be reachable on port 8000, but Linda in namespace 2 is already using that port.

Any clarification, potential solutions, or help in general will be much appreciated.

old_timer
  • 69,149
  • 8
  • 89
  • 168
Ross Peoples
  • 903
  • 2
  • 13
  • 20
  • Perhaps clarify in your question whether these services use the same application protocol or are different and can't be made compatible; around how many distinct services (each with their own tcp/udp) do you expect to support; and do you have any load balancing infrastructure already. – Jonah Benton Mar 05 '18 at 14:52
  • We neither know, nor want to know, what applications/protocols/ports the end users want to run on the platform. Each namespace should be completely oblivious of the fact that other namespaces exist. Very similar to how your AWS user account has no knowledge of mine, nor do we need to consider sharing ports or IP addresses. – Ross Peoples Mar 05 '18 at 15:45

1 Answers1

0

The github issue is an interesting read, and there are some clever workarounds like starting with HTTPS then using ALPN to switch to a custom protocol: https://github.com/kubernetes/kubernetes/issues/23291 but of course then your clients need to know how to do that.

But if the protocols for these TCP and UDP services using the same port are different and don't have a way to interoperate, then the ingress controller needs to be able to allocate the equivalent of a distinct routable IP address- either with the cloud provider, or with the proprietary infrastructure, however that is handled, per exposed service.

I have not looked closely though my sense is that the packaged ingress controllers from nginx and haproxy are not going to have that automation. It would have to be built in coordination with the available infrastructure automation.

Jonah Benton
  • 3,598
  • 1
  • 16
  • 27
  • Since this is an on-premise, bare-metal Kubernetes cluster, I kind of figured there would have to be some sort of automation within the infrastructure. I'm hoping there's someone else out there that's somehow solved this and can at least point me in the right direction. – Ross Peoples Mar 05 '18 at 14:01
  • There are really so many different possibilities, would be helpful to add more color to the question. This thread may also be of interest: https://github.com/kubernetes/ingress-nginx/issues/17 – Jonah Benton Mar 05 '18 at 14:53