IPv6 iptables/nftables multi-wan NETMAP can't access internal hosts from WAN2, WAN3

Question

I have an ip6tables router with 3 WANs, each supporting IPv6. For some reason incoming connections only work for the default route, not ISPs #2 and #3. For example, I can ping the translated IP for WAN1, but the translated IPs for WANs 2 and 3 time out unless I change the default route to go out WAN 2 or 3... one at a time. Outgoing connections work fine and I can policy-route through the different ISPs.

Incoming connections to a ULA IP over WAN2 (translated) should then go back out WAN2, but instead they go out WAN1 (the default gw). This causes all incoming connections to fail over WANs 2 + 3.

I have policy routing set up which works great for outgoing connections.

iptables script

$IP6TABLES -t nat -A POSTROUTING -s $IPv6_ULA -o eno2 -j NETMAP --to $SPECTRUM_IPv6_PD
$IP6TABLES -t nat -A PREROUTING -d $SPECTRUM_IPv6_PD -i eno2 -j NETMAP --to $IPv6_ULA

$IP6TABLES -t nat -A POSTROUTING -s $IPv6_ULA -o he-ipv6-vz -j NETMAP --to $HE_VZ_IPv6
$IP6TABLES -t nat -A PREROUTING -d $HE_VZ_IPv6 -i he-ipv6-vz -j NETMAP --to $IPv6_ULA

$IP6TABLES -t nat -A POSTROUTING -s $IPv6_ULA -o he-ipv6-nw -j NETMAP --to $HE_NW_IPv6
$IP6TABLES -t nat -A PREROUTING -d $HE_NW_IPv6 -i he-ipv6-nw -j NETMAP --to $IPv6_ULA

ip -6 rule

0:  from all lookup local
208:    from all fwmark 0x68 lookup 51820
209:    from all fwmark 0x70 lookup NW
210:    from all fwmark 0x6f lookup SPC
211:    from all fwmark 0x6e lookup VZ
212:    from all fwmark 0x68 lookup 51820
213:    from all to fd8a:9ae9:9ec8:b00::/56 lookup main
214:    from 2001:120:0f06:b48::2/64 lookup NW
215:    from 2001:120:0f07:b48::/64 lookup NW
216:    from 2001:120:88b6::/48 lookup NW
217:    from 2001:120:9f06:242::2/64 lookup VZ
218:    from 2001:120:6f07:242::/64 lookup VZ
219:    from 2001:120:3935::/48 lookup VZ
220:    from all lookup 220
32767:  from all lookup main

ip route

default via fe80::117:30ff:1e9c:b596 dev eno2 proto ra metric 20 mtu 1500 pref medium
default dev he-ipv6-vz metric 100 pref medium
default dev he-ipv6-nw metric 200 pref medium

But accessing the translated IPv6 IPs for internal hosts only work when I change the default ipv6 route on the router. What am I missing?

It's as if I need some kind of connection tracking but I am not sure how to set that up?

ip -6 route show table all

default via 2001:201:021f:242::1 dev he-ipv6-vz table VZ metric 1024 pref medium
default via 2001:201:021f:b48::1 dev he-ipv6-nw table NW metric 1024 pref medium
default dev wg1 table 51820 metric 1024 pref medium
::1 dev lo proto kernel metric 256 pref medium
2001:201:021f:242::/64 dev he-ipv6-vz proto kernel metric 256 pref medium
2001:201:021f:b48::/64 dev he-ipv6-nw proto kernel metric 256 pref medium
2602:6012:1600:ae00::/64 dev eno4 proto dhcp metric 207 pref medium
2602:6012:1600:ae01::/64 dev eno4 proto dhcp metric 207 pref medium
unreachable 2602:6012:1600:ae00::/56 dev lo proto dhcp metric 201 pref medium
2603:2020:426:b::/64 dev eno2 proto ra metric 20 mtu 1500 pref medium
2603:2020:725:7::/64 dev eno2 proto ra metric 20 mtu 1500 pref medium
2603:2020:200:8a8::/64 dev eno2 proto ra metric 20 mtu 1500 pref medium
2603:2020:c25:6::/64 dev eno2 proto ra metric 20 mtu 1500 pref medium
2603:2020:bfc0:10a::/64 dev eno2 proto ra metric 20 mtu 1500 pref medium
fc00:bbbb:bbbb:bb01::8:9a1 dev wg1 proto kernel metric 256 pref medium
fd8a:9ae9:9ec8:b00::/64 dev enp2s0f0 proto kernel metric 256 pref medium
fd8a:9ae9:9ec8:b01::/64 dev wg0 proto kernel metric 256 pref medium
fe80::/64 dev eno1 proto kernel metric 256 pref medium
fe80::/64 dev eno3 proto kernel metric 256 pref medium
fe80::/64 dev eno4 proto kernel metric 256 pref medium
fe80::/64 dev eno2 proto kernel metric 256 pref medium
fe80::/64 dev he-ipv6-nw proto kernel metric 256 pref medium
fe80::/64 dev he-ipv6-vz proto kernel metric 256 pref medium
fe80::/64 dev ifb2 proto kernel metric 256 pref medium
fe80::/64 dev ifb0 proto kernel metric 256 pref medium
fe80::/64 dev ifb1 proto kernel metric 256 pref medium
fe80::/64 dev tun0 proto kernel metric 256 pref medium
fe80::/64 dev enp2s0f0 proto kernel metric 256 pref medium
default via fe80::217:10ff:fe9c:b096 dev eno2 proto ra metric 20 mtu 1500 pref medium
default dev he-ipv6-vz metric 100 pref medium
default dev he-ipv6-nw metric 200 pref medium
default dev wg1 metric 1000 pref medium
local ::1 dev lo table local proto kernel metric 0 pref medium
anycast 2001:201:021f:242:: dev he-ipv6-vz table local proto kernel metric 0 pref medium
local 2001:201:021f:242::2 dev he-ipv6-vz table local proto kernel metric 0 pref medium
anycast 2001:201:021f:b48:: dev he-ipv6-nw table local proto kernel metric 0 pref medium
local 2001:201:021f:b48::2 dev he-ipv6-nw table local proto kernel metric 0 pref medium
anycast 2602:6012:1600:ae00:: dev eno4 table local proto kernel metric 0 pref medium
local 2602:6012:1600:ae00::1 dev eno4 table local proto kernel metric 0 pref medium
anycast 2602:6012:1600:ae01:: dev eno4 table local proto kernel metric 0 pref medium
local 2602:6012:1600:ae01::1 dev eno4 table local proto kernel metric 0 pref medium
local 2603:2020:bfc0:10a:f0dd:bdbf:204c:2f8b dev eno2 table local proto kernel metric 0 pref medium
local fc00:bbbb:bbbb:bb01::8:9a1 dev wg1 table local proto kernel metric 0 pref medium
anycast fd8a:9ae9:9ec8:b00:: dev enp2s0f0 table local proto kernel metric 0 pref medium
local fd8a:9ae9:9ec8:b00::1 dev enp2s0f0 table local proto kernel metric 0 pref medium
anycast fd8a:9ae9:9ec8:b01:: dev wg0 table local proto kernel metric 0 pref medium
local fd8a:9ae9:9ec8:b01::1 dev wg0 table local proto kernel metric 0 pref medium
anycast fe80:: dev eno1 table local proto kernel metric 0 pref medium
anycast fe80:: dev eno2 table local proto kernel metric 0 pref medium
anycast fe80:: dev eno3 table local proto kernel metric 0 pref medium
anycast fe80:: dev eno4 table local proto kernel metric 0 pref medium
anycast fe80:: dev he-ipv6-nw table local proto kernel metric 0 pref medium
anycast fe80:: dev he-ipv6-vz table local proto kernel metric 0 pref medium
anycast fe80:: dev ifb2 table local proto kernel metric 0 pref medium
anycast fe80:: dev ifb0 table local proto kernel metric 0 pref medium
anycast fe80:: dev ifb1 table local proto kernel metric 0 pref medium
anycast fe80:: dev tun0 table local proto kernel metric 0 pref medium
anycast fe80:: dev enp2s0f0 table local proto kernel metric 0 pref medium
local fe80::44e9:b312 dev he-ipv6-nw table local proto kernel metric 0 pref medium
local fe80::4769:fe4d dev he-ipv6-vz table local proto kernel metric 0 pref medium
local fe80::1931:913f:5bdb:289 dev tun0 table local proto kernel metric 0 pref medium
local fe80::3080:ac4d:a9f0:280b dev enp2s0f0 table local proto kernel metric 0 pref medium
local fe80::54f1:e8ae:ae4e:4a25 dev eno3 table local proto kernel metric 0 pref medium
local fe80::7b36:38b8:7a46:6cdf dev ifb2 table local proto kernel metric 0 pref medium
local fe80::9805:bc53:7a4e:1ca7 dev eno2 table local proto kernel metric 0 pref medium
local fe80::a95b:cdbd:dc52:3426 dev eno4 table local proto kernel metric 0 pref medium
local fe80::bf8a:d6dd:95d0:f485 dev eno1 table local proto kernel metric 0 pref medium
local fe80::c0b1:beff:fe11:e8ae dev ifb2 table local proto kernel metric 0 pref medium
local fe80::dbb0:8340:e61e:70d6 dev ifb1 table local proto kernel metric 0 pref medium
local fe80::f290:6345:989a:770d dev ifb0 table local proto kernel metric 0 pref medium
multicast ff00::/8 dev enp2s0f0 table local proto kernel metric 256 pref medium
multicast ff00::/8 dev eno1 table local proto kernel metric 256 pref medium
multicast ff00::/8 dev eno2 table local proto kernel metric 256 pref medium
multicast ff00::/8 dev eno3 table local proto kernel metric 256 pref medium
multicast ff00::/8 dev eno4 table local proto kernel metric 256 pref medium
multicast ff00::/8 dev he-ipv6-nw table local proto kernel metric 256 pref medium
multicast ff00::/8 dev he-ipv6-vz table local proto kernel metric 256 pref medium
multicast ff00::/8 dev ifb2 table local proto kernel metric 256 pref medium
multicast ff00::/8 dev wg0 table local proto kernel metric 256 pref medium
multicast ff00::/8 dev ifb0 table local proto kernel metric 256 pref medium
multicast ff00::/8 dev ifb1 table local proto kernel metric 256 pref medium
multicast ff00::/8 dev tun0 table local proto kernel metric 256 pref medium
multicast ff00::/8 dev wg1 table local proto kernel metric 256 pref medium

I'm not sure exactly what's going on here, but I think there's likely enough information now for someone more experienced than me to take a stab at it. I did want to note that [we recommend against obfuscation](https://meta.serverfault.com/q/963/126632) whenever possible, and especially for _private_ IP addresses (which it seems you also obfuscated) as that can be confusing. — Michael Hampton, Feb 12 '21 at 15:38
my 2¢: https://home.regit.org/netfilter-en/netfilter-connmark/ (but OP doesn't tell where the fwmark routing entries get their mark from. Missing rules?) — A.B, Mar 03 '21 at 16:29

IPv6 iptables/nftables multi-wan NETMAP can't access internal hosts from WAN2, WAN3

0 Answers0