0

After long searching I've found the answer to my challenge, but the last step fails.

I have two identical servers that both serve multiple guests. The servers have two interfaces: one for the external IP's I got from my ISP and one for an internal back-end network. All guests get an internal back-end network address and are connected to the physical back-end network via a bridge. Both servers can ping all guests on both servers. I expose the guests by D/SNAT'ing the guest IP to one of the available IP's on the host. This works perfectly for guests that live on the host.

For the D/SNAT to work to the other host, I learned that I needed to add a route for packets that arrived on the back-end interface, because the default route told the stack on the other server to go back on the wrong interface and wrong public IP. I have this scenario working in two local virtual test machines, even over a "dummy" bridge to the physical back-end interface.

This also works on the real servers for DNAT'ing to the back-end IP address that lives on the back-end bridge but NOT for the addresses assigned to the guests on the other server, which are also connected to the same back-end bridge.

Configuration, server A:

/etc/iproute2/rt_tables:
..
200 internal

iptables -P FORWARD ACCEPT
iptables -t nat -A POSTROUTING -o $WAN_IF -j SNAT --to $PUBLIC_IP
iptables -t nat -A PREROUTING -i $WAN_IF -p tcp -d $PUBLIC_IP --dport 53 -j DNAT --to $GUEST_IP
ip rule add from $LAN_IP table internal
ip route add default via $LAN_GW dev $LAN_IF table internal

Where
WAN_IF = WAN bridge interface (br1)
PUBLIC_IP = is public ip address on WAN bridge if
LAN_IP = private ip on LAN bridge (br0)
LAN_GW = private ip on LAN bridge on other server

brctl show
br0             8000.003018a96c83       no              eth0
                                                        vnet0
                                                        vnet1
                                                        vnet2
                                                        vnet3
                                                        vnet4
                                                        vnet5
br1             8000.003018a96c84       no              eth1

When $GUEST_IP is ip of guest on same host, everything works. When $GUEST_IP is LAN ip of other server also. But I can't reach guests on other server?

Update

Adjusting the default gateway of the guest works, but that isn't a symmetric solution as I wanted to create. But if that needs to be, it's a small change to make things work, as long as I keep track of the routes. It would however render routing different protocols via different public IP's impossible though unless I create some rules for that on the guest as well.

Update 2

It turns out I can solve the problem by adjusting the default gateway of the guest alone, but this is not as 'flexible' as I had hoped for.

Is there a more elegant/flexible solution?

user45156
  • 41
  • 2
  • you could try looking at the traffic on the bridge tcpdump -i br0 host $GUEST_IP and print you route table, is it routing via the bridge or via the interface, and which interface owns the IP eth0 or br0, is there a VLAN tag between the two hosts? – Sum1sAdmin Apr 13 '16 at 09:40
  • also ping -I br0 $OTHER_BRIDGE – Sum1sAdmin Apr 13 '16 at 09:42
  • All addresses are assigned to the bridge interfaces, not the physical if's. I have one idea I haven't thought of: all guests have default gateway to the hosts $LAN_IP, which would render the above scenario broken, because the returning packets of the guest would be routed to the wrong host. – user45156 Apr 13 '16 at 11:18
  • Removing the default gateway on a guest doesn't help. – user45156 Apr 13 '16 at 11:35
  • Setting other host as default gateway solves the problem. But that isn't a satisfactory solution, because then all requests via the host would fail I guess. – user45156 Apr 13 '16 at 11:47
  • the Bridge IP is a LAN ip yes? - the guests get D/NAT'ed to the LAN ip and the NAT table should route back to guests - if both bridges can ping each other then there's something up with NAT Something like iptables -t nat -A POSTROUTING -o eth1 -j MASQUERADE iptables -A FORWARD -i br0 -o eth1 -j ACCEPT – Sum1sAdmin Apr 13 '16 at 11:49
  • I don't want to masquerade because I want to preserve external IP in logs. That's what makes this a thorny routing challenge. – user45156 Apr 13 '16 at 13:26
  • so the switch port needs to be a trunk and not in access mode for any vlan - can you plug any vlan into the switch port? – Sum1sAdmin Apr 13 '16 at 13:30
  • There is no switch, both servers are directly connected. Please read up on my update. I can fix the problem by changing default gateway in guest, but I was hoping for a more universal solution that wouldn't require changing the guest IP conf. – user45156 Apr 13 '16 at 14:11
  • have you looked at open virtual switch? http://openvswitch.org/support/config-cookbooks/vlan-configuration-cookbook/ – Sum1sAdmin Apr 13 '16 at 14:58

1 Answers1

0

Adjusting the default gateway of the guest seems to be te simplest solution.

user45156
  • 41
  • 2