2

My current setup involves an EKS Cluster with multiple namespaces (multi-tenant) across many different EKS nodes in private subnets. I would like the egress traffic from the pods to have a dedicated EIP per namespace. AFAIK there are no off the shelf solutions available for this problem. I have searched long and hard on the internet but in vain. Here are some of the solutions that I have tired, but eventually hit a roadblock

  1. Istio Egress Gateway Allows you to direct all outbound traffic from pods via a dedicated egress gateway pod. But the egress gateway pod will assume the ip of the node on which it is running, which will not work for my use case. Also i have not found decent documentation around how i can setup multiple egress gateway pods across multiple nodes.

  2. Calico Egress Gateway Very similar to Istio Egress Gateway solution and the same constraint applies

  3. Custom Solution I have started implementing a custom solution, where I setup multiple Gateway Nodes in public subnet each with a dedicated EIP. I can now modify the ip routes/gateways in the private EKS node, to route traffic via a specific gateway node based on the pod source ip. This solution feels very kludgy and the operational overhead of such a solution is very high

  4. I have looked at solutions like https://github.com/nirmata/kube-static-egress-ip, but have not had any luck with them

Is there a better approach/solution to this problem?

p.s: My production cluster is extremely large, I cannot afford to stand up a cluster for each namespace.

Mark Rotteveel
  • 100,966
  • 191
  • 140
  • 197
Karthik Balasubramanian
  • 1,127
  • 4
  • 13
  • 36
  • I have cross posted this question on serverfault (https://serverfault.com/questions/1082748/how-can-i-get-egress-static-ip-per-namespace-within-a-eks-cluster) as well. Happy to delete this question here if it is not appropriate – Karthik Balasubramanian Nov 05 '21 at 19:30
  • I see your issue has its end, would you close your question in serverfault or give there a brief summary answer based on the comments here? – Bazhikov Nov 09 '21 at 17:36

1 Answers1

0

I thought that Calico Enterprise would solve for this specific use case by assigning an IP to the namespace. From this blog: "[Egress Gateway] defines a static egress IP for SNAT on traffic leaving the cluster and applies it to a specific namespace. Then it designates one pod in a namespace as the egress pod and assigns a routable IP which is used specifically for egress traffic leaving the cluster."

Do you mind also +1 this feature request? https://github.com/aws/containers-roadmap/issues/1319

mreferre
  • 5,464
  • 3
  • 22
  • 29
  • Thanks. I have reached out to calico folks. Their support for egress eip is in the works. Am just surprised that this is not a common problem and a solution for this doest not exist yet – Karthik Balasubramanian Nov 06 '21 at 02:55
  • For my personal curiosity can you elaborate more? I haven't tested Calico Egress Gateway but from the docs it seems that they do already have namespace-based egress ip as a product feature. Or do you need a more sophisticated egress setup that they can't do (yet)? Thanks. – mreferre Nov 06 '21 at 08:48
  • As I understand it, egress gateway pod is not aws vps aware, hence it cannot assign EIPs to egress pods. When the traffic exits out of gateway pods it assumes the ip of the Node itself as the source ip which is not what I want. – Karthik Balasubramanian Nov 06 '21 at 14:49
  • Thanks. I thought that it being applied to a specific namaspace it entailed it was abstracted from the infrastructure and where pods were deployed but, as I said, I have not used it so I am just relying on what I am reading in their docs. – mreferre Nov 06 '21 at 16:28
  • @karthik, `many different EKS nodes in private subnets...` EIP does not work in private subnet. If you have worker nodes designated in public subnet, you can use EIP and reverse proxy on these public accessible nodes to expose services in private network. You can then use Calico Egress Gateway on these public nodes for the reverse proxy pods in turn use the underlying EIP. – gohm'c Nov 07 '21 at 08:29