-1

say I have 2 sites, S1,S2; each with at least a kubernetes worker. The two sites are geographically apart and have different public IPs on the nodes/workers.

Does kubernetes offer any existing mechanisms to route outgoing internet traffic from a pod/container in S1, via S2 ?

The goal is to be able to use the public IP(s) in S2 for pods in S1.

If k8s-federation is a requisite for a solution; then that is fine.

hbogert
  • 4,198
  • 5
  • 24
  • 38

2 Answers2

1

Kubernetes doesn't have any input on this, it's up to your network design and structure, just like it would be with traditional VMs or whatnot. That said, this sounds like a very bad network design given what you described so I would be surprised if it was easy to set up. Calico runs normal BGP under the hood so you can probably set up two ASes and force one to route via the other.

coderanger
  • 52,400
  • 4
  • 52
  • 75
  • Maybe I should've emphasized this, but the custom routing should be per pod. IIUC your solution would have all traffic go from S1 to S2 – hbogert Dec 31 '18 at 10:08
  • 1
    Again, speaking only for Calico, you can create multiple IP groups and assign a pod to a non-default one through an annotation. That could allow for this with some very careful network design. – coderanger Dec 31 '18 at 12:07
0

With calico you can introduce a non default IPPool which does not do NAT for outgoing traffic. Then you can annotate the pods and/or namespaces you want to use that IPPool. At that point it leaves you with non-working outgoing traffic, since your cluster IPs are "leaking" to the outside world. But a return path is not known by any upstream routers between your cluster and the internet.

You have to let your router, which should be between your cluster and the internet, know about the cluster ranges. You can use the global BGPPeer concept from calico. Once you've done that, als set up BGP on your router. (Use [1]) for more info.

From there you should have all the flexibility on your router to route it differently based on the non default IPPool's subnet and/or first tunnel it, e.g., to the questioner's 'S2'

Note that unless you are truly using public IP space, i.e., non RFC-1918 IPs (plus some others), you should introduce NATing somewhere yourself now, you can choose to do so in S1 or S2, if you opt for the latter, than that site also needs to know about the return path back to your router.

This is not really a cloud-native solution since you're just moving the problem from kubernetes to "old-school" domain of policy based routing on fixed subnets -- which is not really what the questioner asked for since he implied that there is also a kubernetes process in 'S2'. In the possible solution above, a k8s process is not needed in S2.

This is what @coderanger's custom outgoing network path for kubernetes pod was suggesting

hbogert
  • 4,198
  • 5
  • 24
  • 38
  • partial needed functionality is now also part of calico enterprise. https://www.tigera.io/features/egress-gateway/ – hbogert Aug 07 '20 at 15:03