0

I have a system with two network interfaces, both connecting to same default gateway. I want to load balance the connections to the same destination IP made from the system across both interfaces and use source IPs from each. I want this solution to be generic and work for whatever program we run in the system. What I've observed till now, is that linux maintains a mini routing cache, so once the first few connections are established, locks on to one route.. so the initial few connections are distributed across source IPs of two interfaces, but ultimately converges to one of them(I'm guessing there's some logic to lock on to deciding which route is better than other in case of conflicts)

Using iptables will influence the packet only after the initial routing decision is made and source IP is selected. Is there any way whatsoever to influence the source IP selection of socket to load balance across multiple interfaces?

UPDATE: I came across eBPF that can modify socket properties, can this be leveraged to modify source address in round robin across all available interfaces (except loopback)

TheJoker
  • 21
  • 3
  • I would suggest load balancing on L2 with interface bonding or teaming. What you try to build is L3 balancing and it is very hard. By default linux kernel even can't recognize input IP address to throw packet via second interface. – kab00m Apr 30 '21 at 09:07
  • @kab00m L2 load balancer won't work for us, the whole purpose of this to enable every process to create more than 64k connections to same destination IP – TheJoker Apr 30 '21 at 10:43
  • @TheJoker the problem is that you try to fix an application. If your application (described in Q/A: https://serverfault.com/questions/1061423/multipath-routing-for-outbound-connection-with-source-address-as-0-0-0-0 ) was binding to multiple addresses, you wouldn't have to ask all these questions. – A.B Apr 30 '21 at 10:52
  • Also Linux >= 3.6 doesn't use a routing cache anymore (except special cases like pmtud or redirect) for IPv4. What is your kernel? – A.B Apr 30 '21 at 10:59
  • @A.B we need it supporting any and all applications! Yes, fixing for each application one by one is an option Kernel version is 5+. Routing cache was added back at a later point in time I believe. Route is definitely cached in fib after opening more than ~2000 connections as all further connections stick to one interface – TheJoker Apr 30 '21 at 11:09
  • just about routing cache; `ip route show cache`. If it displays something tell me, I'm curious. – A.B Apr 30 '21 at 11:11
  • @A.B no it doesn't show anything. but `ip route get to `, after flushing, will round robin between two interfaces. After opening more than 2000 connections, will stick to one interface and ip – TheJoker Apr 30 '21 at 11:14
  • These are different tasks - load balancing across physical interfaces and initiating connection from different IPs. If your connections are incoming you can just use L2 balancing and two IP on one bond interface. Does that work for you? – kab00m Apr 30 '21 at 12:29
  • @kab00m unfortunately no. The connections are not incoming, they're generated within the machine.. I know we could've configured fwmark and prerouting chains for incoming connection but that can't work in my case. Looks like writing a kernel module is the only way forward.. – TheJoker Apr 30 '21 at 18:10

0 Answers0