0

I have a server behind two public facing gateways 

INTERNET
 |   |
GW1 GW2
 |   |
 SERVER

GW1's IP is 192.168.w.x, MAC aa:bb:cc:dd:ee:ff
GW2's IP is 192.168.y.z, MAC ff:ee:dd:cc:bb:aa # note: w != y

and I'm trying to load balance outgoing connections from the server. I've built upon pepoluan's solution here and mefat's solution here, but those are for dealing with incoming connections.

Here's my /etc/iproute2/rt_tables

...
1   gw1
2   gw2

I setup iproute2 routes/rules as

ip route add default via 192.168.w.x table gw1
ip route add default via 192.168.y.z table gw2 # w != y
ip rule add fwmark 1 table gw1
ip rule add fwmark 2 table gw2

The iptables rules on the server are

iptables -t mangle -A PREROUTING -j CONNMARK --restore-mark
iptables -t mangle -A PREROUTING -m mark '!' --mark 0 -j ACCEPT

# incoming (ensure that responses are routed back through the GW from which they came)
iptables -t mangle -A PREROUTING -m mac --mac-source aa:bb:cc:dd:ee:ff -j MARK --set-mark 0x1
iptables -t mangle -A PREROUTING -m mac --mac-source ff:ee:dd:cc:bb:aa -j MARK --set-mark 0x2

# outgoing (load balance)
iptables -t mangle -A OUTPUT -m statistic --mode nth --every 2 --packet 0 -j MARK --set-mark 0x1
iptables -t mangle -A OUTPUT -m statistic --mode nth --every 2 --packet 1 -j MARK --set-mark 0x2

# save mark
iptables -t mangle -A POSTROUTING -j CONNMARK --save-mark

I've masqueraded the private interface to the public interface on each of the gateways and have the server as a trusted machine on the gateways so the gateways accept incoming connections from it.

When I do

ping -m 1 8.8.4.4

on the server I can see that the packets have the 0x1 mark set and they go from OUTPUT to nat:POSTROUTING but don't seem to make it to the gateway.

kernel: TRACE: raw:OUTPUT:policy:3 IN= OUT=eth0 SRC=192.168.a.b DST=8.8.4.4 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=26835 DF PROTO=ICMP TYPE=8 CODE=0 ID=2004 SEQ=1 UID=0 GID=0 MARK=0x1 
kernel: TRACE: mangle:OUTPUT:rule:1 IN= OUT=eth0 SRC=192.168.a.b DST=8.8.4.4 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=26835 DF PROTO=ICMP TYPE=8 CODE=0 ID=2004 SEQ=1 UID=0 GID=0 MARK=0x1 
kernel: TRACE: mangle:OUTPUT_direct:return:1 IN= OUT=eth0 SRC=192.168.a.b DST=8.8.4.4 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=26835 DF PROTO=ICMP TYPE=8 CODE=0 ID=2004 SEQ=1 UID=0 GID=0 MARK=0x1 
kernel: TRACE: mangle:OUTPUT:policy:2 IN= OUT=eth0 SRC=192.168.a.b DST=8.8.4.4 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=26835 DF PROTO=ICMP TYPE=8 CODE=0 ID=2004 SEQ=1 UID=0 GID=0 MARK=0x1 
...
kernel: TRACE: nat:POST_public_allow:return:1 IN= OUT=eth0 SRC=192.168.a.b DST=8.8.4.4 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=26835 DF PROTO=ICMP TYPE=8 CODE=0 ID=2004 SEQ=1 UID=0 GID=0 MARK=0x1 
kernel: TRACE: nat:POST_public:return:4 IN= OUT=eth0 SRC=192.168.a.b DST=8.8.4.4 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=26835 DF PROTO=ICMP TYPE=8 CODE=0 ID=2004 SEQ=1 UID=0 GID=0 MARK=0x1 
kernel: TRACE: nat:POSTROUTING:policy:4 IN= OUT=eth0 SRC=192.168.a.b DST=8.8.4.4 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=26835 DF PROTO=ICMP TYPE=8 CODE=0 ID=2004 SEQ=1 UID=0 GID=0 MARK=0x1

(The chains like "POST_public_allow" are a vestige of the fact that I'm actually using firewalld for the iptables rules.)

What am I doing wrong or missing? Do I need a FORWARD rule? How can I debug iproute2 routes/rules?

EDIT: added output as requested by @A.B

Apologies for the noise introduced by firewalld.

[root@server]# ip -br link
lo               UNKNOWN        00:00:00:00:00:00 <LOOPBACK,UP,LOWER_UP>
eth0             UP             ee:aa:bb:ee:ee:ff <BROADCAST,MULTICAST,UP,LOWER_UP>

[root@server]# ip -4 -br address
lo               UNKNOWN        127.0.0.1/8
eth0             UP             192.168.a.b/17 i.j.k.l/24

[root@server]# ip rule
0:  from all lookup local
32764:  from all fwmark 0x2 lookup gw2
32765:  from all fwmark 0x1 lookup gw1
32766:  from all lookup main
32767:  from all lookup default

[root@server]# ip route
default via i.j.k.1 dev eth0 proto static metric 100
i.j.k.0/24 dev eth0 proto kernel scope link src i.j.k.l metric 100
192.168.128.0/17 dev eth0 proto kernel scope link src 192.168.a.b metric 100

[root@server]# ip route show table gw1
default via 192.168.w.x dev eth0

[root@server]#  ip route show table gw2
default via 192.168.y.z dev eth0

[root@server]# iptables-save -c
# Generated by iptables-save v1.6.2 on Wed Jul 11 19:44:43 2018
*nat
:PREROUTING ACCEPT [234747:12721357]
:INPUT ACCEPT [22:1268]
:OUTPUT ACCEPT [855:88393]
:POSTROUTING ACCEPT [526:64778]
:OUTPUT_direct - [0:0]
:POSTROUTING_ZONES - [0:0]
:POSTROUTING_ZONES_SOURCE - [0:0]
:POSTROUTING_direct - [0:0]
:POST_public - [0:0]
:POST_public_allow - [0:0]
:POST_public_deny - [0:0]
:POST_public_log - [0:0]
:POST_trusted - [0:0]
:POST_trusted_allow - [0:0]
:POST_trusted_deny - [0:0]
:POST_trusted_log - [0:0]
:PREROUTING_ZONES - [0:0]
:PREROUTING_ZONES_SOURCE - [0:0]
:PREROUTING_direct - [0:0]
:PRE_public - [0:0]
:PRE_public_allow - [0:0]
:PRE_public_deny - [0:0]
:PRE_public_log - [0:0]
:PRE_trusted - [0:0]
:PRE_trusted_allow - [0:0]
:PRE_trusted_deny - [0:0]
:PRE_trusted_log - [0:0]
[234747:12721357] -A PREROUTING -j PREROUTING_direct
[234747:12721357] -A PREROUTING -j PREROUTING_ZONES_SOURCE
[234747:12721357] -A PREROUTING -j PREROUTING_ZONES
[855:88393] -A OUTPUT -j OUTPUT_direct
[855:88393] -A POSTROUTING -j POSTROUTING_direct
[526:64778] -A POSTROUTING -j POSTROUTING_ZONES_SOURCE
[526:64778] -A POSTROUTING -j POSTROUTING_ZONES
[524:64548] -A POSTROUTING_ZONES -o eth0 -g POST_public
[2:230] -A POSTROUTING_ZONES -g POST_public
[2:230] -A POSTROUTING_ZONES_SOURCE -m set --match-set trusted_hosts dst -j POST_trusted
[329:23615] -A POSTROUTING_direct -m mark ! --mark 0x0 -j SNAT --to-source 192.168.a.b
[526:64778] -A POST_public -j POST_public_log
[526:64778] -A POST_public -j POST_public_deny
[526:64778] -A POST_public -j POST_public_allow
[2:230] -A POST_trusted -j POST_trusted_log
[2:230] -A POST_trusted -j POST_trusted_deny
[2:230] -A POST_trusted -j POST_trusted_allow
[234747:12721357] -A PREROUTING_ZONES -i eth0 -g PRE_public
[0:0] -A PREROUTING_ZONES -g PRE_public
[3:180] -A PREROUTING_ZONES_SOURCE -m set --match-set trusted_hosts src -j PRE_trusted
[234747:12721357] -A PRE_public -j PRE_public_log
[234747:12721357] -A PRE_public -j PRE_public_deny
[234747:12721357] -A PRE_public -j PRE_public_allow
[3:180] -A PRE_trusted -j PRE_trusted_log
[3:180] -A PRE_trusted -j PRE_trusted_deny
[3:180] -A PRE_trusted -j PRE_trusted_allow
COMMIT
# Completed on Wed Jul 11 19:44:43 2018
# Generated by iptables-save v1.6.2 on Wed Jul 11 19:44:43 2018
*mangle
:PREROUTING ACCEPT [623828:32865698]
:INPUT ACCEPT [623836:32866210]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [153676:12802286]
:POSTROUTING ACCEPT [534555:70869287]
:FORWARD_direct - [0:0]
:INPUT_direct - [0:0]
:OUTPUT_direct - [0:0]
:POSTROUTING_direct - [0:0]
:PREROUTING_ZONES - [0:0]
:PREROUTING_ZONES_SOURCE - [0:0]
:PREROUTING_direct - [0:0]
:PRE_public - [0:0]
:PRE_public_allow - [0:0]
:PRE_public_deny - [0:0]
:PRE_public_log - [0:0]
:PRE_trusted - [0:0]
:PRE_trusted_allow - [0:0]
:PRE_trusted_deny - [0:0]
:PRE_trusted_log - [0:0]
[623836:32866210] -A PREROUTING -j PREROUTING_direct
[623828:32865698] -A PREROUTING -j PREROUTING_ZONES_SOURCE
[623828:32865698] -A PREROUTING -j PREROUTING_ZONES
[623836:32866210] -A INPUT -j INPUT_direct
[0:0] -A FORWARD -j FORWARD_direct
[534523:70856073] -A OUTPUT -j OUTPUT_direct
[534555:70869287] -A POSTROUTING -j POSTROUTING_direct
[154527:12862895] -A OUTPUT_direct -j CONNMARK --restore-mark --nfmask 0xffffffff --ctmask 0xffffffff
[851:60609] -A OUTPUT_direct -m mark ! --mark 0x0 -j ACCEPT
[76838:6400954] -A OUTPUT_direct -m statistic --mode nth --every 2 --packet 0 -j MARK --set-xmark 0x1/0xffffffff
[76838:6401332] -A OUTPUT_direct -m statistic --mode nth --every 2 --packet 1 -j MARK --set-xmark 0x2/0xffffffff
[534555:70869287] -A POSTROUTING_direct -j CONNMARK --save-mark --nfmask 0xffffffff --ctmask 0xffffffff
[623792:32858768] -A PREROUTING_ZONES -i eth0 -g PRE_public
[36:6930] -A PREROUTING_ZONES -g PRE_public
[378266:19709657] -A PREROUTING_ZONES_SOURCE -m set --match-set trusted_hosts src -j PRE_trusted
[623836:32866210] -A PREROUTING_direct -j CONNMARK --restore-mark --nfmask 0xffffffff --ctmask 0xffffffff
[8:512] -A PREROUTING_direct -m mark ! --mark 0x0 -j ACCEPT
[0:0] -A PREROUTING_direct -s 192.168.w.x/32 -j MARK --set-xmark 0x1/0xffffffff
[0:0] -A PREROUTING_direct -s 192.168.y.z/32 -j MARK --set-xmark 0x2/0xffffffff
[623828:32865698] -A PRE_public -j PRE_public_log
[623828:32865698] -A PRE_public -j PRE_public_deny
[623828:32865698] -A PRE_public -j PRE_public_allow
[378266:19709657] -A PRE_trusted -j PRE_trusted_log
[378266:19709657] -A PRE_trusted -j PRE_trusted_deny
[378266:19709657] -A PRE_trusted -j PRE_trusted_allow
COMMIT
# Completed on Wed Jul 11 19:44:43 2018
# Generated by iptables-save v1.6.2 on Wed Jul 11 19:44:43 2018
*raw
:PREROUTING ACCEPT [623836:32866210]
:OUTPUT ACCEPT [534530:70860961]
:OUTPUT_direct - [0:0]
:PREROUTING_ZONES - [0:0]
:PREROUTING_ZONES_SOURCE - [0:0]
:PREROUTING_direct - [0:0]
:PRE_public - [0:0]
:PRE_public_allow - [0:0]
:PRE_public_deny - [0:0]
:PRE_public_log - [0:0]
:PRE_trusted - [0:0]
:PRE_trusted_allow - [0:0]
:PRE_trusted_deny - [0:0]
:PRE_trusted_log - [0:0]
[623836:32866210] -A PREROUTING -j PREROUTING_direct
[623836:32866210] -A PREROUTING -j PREROUTING_ZONES_SOURCE
[623836:32866210] -A PREROUTING -j PREROUTING_ZONES
[534530:70860961] -A OUTPUT -j OUTPUT_direct
[623796:32859080] -A PREROUTING_ZONES -i eth0 -g PRE_public
[40:7130] -A PREROUTING_ZONES -g PRE_public
[378268:19709777] -A PREROUTING_ZONES_SOURCE -m set --match-set trusted_hosts src -j PRE_trusted
[623836:32866210] -A PRE_public -j PRE_public_log
[623836:32866210] -A PRE_public -j PRE_public_deny
[623836:32866210] -A PRE_public -j PRE_public_allow
[378268:19709777] -A PRE_trusted -j PRE_trusted_log
[378268:19709777] -A PRE_trusted -j PRE_trusted_deny
[378268:19709777] -A PRE_trusted -j PRE_trusted_allow
COMMIT
# Completed on Wed Jul 11 19:44:43 2018
# Generated by iptables-save v1.6.2 on Wed Jul 11 19:44:43 2018
*security
:INPUT ACCEPT [378299:19711565]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [534530:70860961]
:FORWARD_direct - [0:0]
:INPUT_direct - [0:0]
:OUTPUT_direct - [0:0]
[378299:19711565] -A INPUT -j INPUT_direct
[0:0] -A FORWARD -j FORWARD_direct
[534530:70860961] -A OUTPUT -j OUTPUT_direct
COMMIT
# Completed on Wed Jul 11 19:44:43 2018
# Generated by iptables-save v1.6.2 on Wed Jul 11 19:44:43 2018
*filter
:INPUT ACCEPT [0:0]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [534519:70858874]
:FORWARD_IN_ZONES - [0:0]
:FORWARD_IN_ZONES_SOURCE - [0:0]
:FORWARD_OUT_ZONES - [0:0]
:FORWARD_OUT_ZONES_SOURCE - [0:0]
:FORWARD_direct - [0:0]
:FWDI_public - [0:0]
:FWDI_public_allow - [0:0]
:FWDI_public_deny - [0:0]
:FWDI_public_log - [0:0]
:FWDI_trusted - [0:0]
:FWDI_trusted_allow - [0:0]
:FWDI_trusted_deny - [0:0]
:FWDI_trusted_log - [0:0]
:FWDO_public - [0:0]
:FWDO_public_allow - [0:0]
:FWDO_public_deny - [0:0]
:FWDO_public_log - [0:0]
:FWDO_trusted - [0:0]
:FWDO_trusted_allow - [0:0]
:FWDO_trusted_deny - [0:0]
:FWDO_trusted_log - [0:0]
:INPUT_ZONES - [0:0]
:INPUT_ZONES_SOURCE - [0:0]
:INPUT_direct - [0:0]
:IN_public - [0:0]
:IN_public_allow - [0:0]
:IN_public_log - [0:0]
:IN_trusted - [0:0]
:IN_trusted_allow - [0:0]
:IN_trusted_deny - [0:0]
:IN_trusted_log - [0:0]
:OUTPUT_direct - [0:0]
[70:20292] -A INPUT -p udp -m multiport --dports 5353 -j ACCEPT
[378201:19689725] -A INPUT -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
[2:120] -A INPUT -i lo -j ACCEPT
[245557:13155833] -A INPUT -j INPUT_direct
[245557:13155833] -A INPUT -j INPUT_ZONES_SOURCE
[245554:13155653] -A INPUT -j INPUT_ZONES
[10812:434556] -A INPUT -m conntrack --ctstate INVALID -j DROP
[234725:12720089] -A INPUT -j REJECT --reject-with icmp-host-prohibited
[0:0] -A FORWARD -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
[0:0] -A FORWARD -i lo -j ACCEPT
[0:0] -A FORWARD -j FORWARD_direct
[0:0] -A FORWARD -j FORWARD_IN_ZONES_SOURCE
[0:0] -A FORWARD -j FORWARD_IN_ZONES
[0:0] -A FORWARD -j FORWARD_OUT_ZONES_SOURCE
[0:0] -A FORWARD -j FORWARD_OUT_ZONES
[0:0] -A FORWARD -m conntrack --ctstate INVALID -j DROP
[0:0] -A FORWARD -j REJECT --reject-with icmp-host-prohibited
[534530:70860961] -A OUTPUT -j OUTPUT_direct
[0:0] -A FORWARD_IN_ZONES -i eth0 -g FWDI_public
[0:0] -A FORWARD_IN_ZONES -g FWDI_public
[0:0] -A FORWARD_IN_ZONES_SOURCE -m set --match-set trusted_hosts src -j FWDI_trusted
[0:0] -A FORWARD_OUT_ZONES -o eth0 -g FWDO_public
[0:0] -A FORWARD_OUT_ZONES -g FWDO_public
[0:0] -A FORWARD_OUT_ZONES_SOURCE -m set --match-set trusted_hosts dst -j FWDO_trusted
[0:0] -A FWDI_public -j FWDI_public_log
[0:0] -A FWDI_public -j FWDI_public_deny
[0:0] -A FWDI_public -j FWDI_public_allow
[0:0] -A FWDI_public -p icmp -j ACCEPT
[0:0] -A FWDI_trusted -j FWDI_trusted_log
[0:0] -A FWDI_trusted -j FWDI_trusted_deny
[0:0] -A FWDI_trusted -j FWDI_trusted_allow
[0:0] -A FWDI_trusted -j ACCEPT
[0:0] -A FWDO_public -j FWDO_public_log
[0:0] -A FWDO_public -j FWDO_public_deny
[0:0] -A FWDO_public -j FWDO_public_allow
[0:0] -A FWDO_trusted -j FWDO_trusted_log
[0:0] -A FWDO_trusted -j FWDO_trusted_deny
[0:0] -A FWDO_trusted -j FWDO_trusted_allow
[0:0] -A FWDO_trusted -j ACCEPT
[245554:13155653] -A INPUT_ZONES -i eth0 -g IN_public
[0:0] -A INPUT_ZONES -g IN_public
[3:180] -A INPUT_ZONES_SOURCE -m set --match-set trusted_hosts src -j IN_trusted
[245554:13155653] -A IN_public -j IN_public_log
[245554:13155653] -A IN_public -j IN_public_deny
[245554:13155653] -A IN_public -j IN_public_allow
[17:1008] -A IN_public -p icmp -j ACCEPT
[0:0] -A IN_public_allow -d 224.0.0.251/32 -p udp -m udp --dport 5353 -m conntrack --ctstate NEW -j ACCEPT
[3:180] -A IN_trusted -j IN_trusted_log
[3:180] -A IN_trusted -j IN_trusted_deny
[3:180] -A IN_trusted -j IN_trusted_allow
[3:180] -A IN_trusted -j ACCEPT
COMMIT
# Completed on Wed Jul 11 19:44:43 2018
pooch
  • 1
  • 3
  • If your gateways are really `192.168.n.n` then I assume this means the routers you are pointing do are doing NAT? This means what you are trying to do will almost certainly NOT work. It would confused the hell out of the NAT, not seeing the full connection, plus any firewalls of whatever you are connecting too wouldn't be happy with with your source ip/port switching between the public IP of the two routers. I suspect the best you can do is balance based on socket, not per-packet. – Zoredache Jul 05 '18 at 21:20
  • Thanks for the reply. You are correct about the gateways doing NAT. However, the marking systems for outgoing connections presented by pepoluan & @mefat guarantee that the public IP seen by machines on the internet is stable, ie connections through GW1 always go through GW1; same for GW2. My problem with the ping seems to be that the server is not connecting to a gateway (GW1, specifically). Using wireshark on the server shows `Echo (ping) request id=0x3b42, seq=1/256, ttl=64 (no response found)`. I thought the ip routing table would route to GW1 given mark 0x1. Do I need a FORWARD rule? – pooch Jul 06 '18 at 07:40
  • @Zoredache most of the magic is handled by CONNMARK, see there for example: https://home.regit.org/netfilter-en/netfilter-connmark/ – A.B Jul 10 '18 at 19:04
  • I can see multiple issues (missing routes in additional routing tables, how many ips on server? , badly placed iptables rules, missing iptables rules...) in your settings, but they can probably be solved. I would need you to be more precise in your settings, and dump these informations (feel free to obfuscate, remove `-br` if it doesn't work with `-br`): `ip -br link; ip -4 -br address; ip rule; ip route; ip route show table gw1; ip route show table gw2; iptables-save -c`. Also: do you intend to load-balance (remotely initiated) incoming too, or is it fine to remove unneeded stuff? – A.B Jul 10 '18 at 19:12
  • @A.B server has 2 ips. See the edit above for the output of the commands that you requested. Load balancing is desired only for outgoing connections. Your help is much appreciated! – pooch Jul 11 '18 at 20:06

1 Answers1

0

Doh! My server is bare metal but the gateways are virtual private servers (VPS) and the VPS provider imposes limitations that drop packets that aren't destined for machines outside of the private network. ie, My network topology will not work due to constraints imposed by my VPS provider.

pooch
  • 1
  • 3
  • I don't get it. Do you mean even with correct settings on the server it can't work anyway? The public IP doesn't have to (nor can) be used at all for outgoing – A.B Jul 13 '18 at 21:22
  • Yes, with correct settings it can't work because, for example, ping -m 1 -c 1 8.8.8.8 will get routed to gw1 but gw1's VPS rules will drop it because VPS will only accept packets with either its private or public IP address (even when forwarding is enabled). That happens because of implicit rules imposed by the VPS provider. – pooch Jul 14 '18 at 08:58