IPv6 Neighbor Discovery/Routing Failing for Kernel 3.10

Question

Stupid mistakes led to all of this, read Update 5

Intro

I'm trying to setup a Linux Router (LXC, 3.10.0-123.el7.x86_64) with IPv6.

The provider is Hetzner and I got 2 subnets, one /56 and a /64. They have setup the routing to my LL Address and my default gateway is fe80::1.

I have enabled ipv6 and ipv4 forwarding in sysctl.conf:

net.ipv4.ip_forward = 1
net.ipv6.conf.all.forwarding = 1

and I configured the IPTables to Accept everything.

I know that ICMPv6 is the main utility used for routing in IPv6 - so that's not blocked ;-). I also added -A FORWARD -j ACCEPT.

I have configured the second (::2) ip of my /56 subnet to eth1 and the second (::2) of the /64 to eth0.

eth0 is the uplink interface
eth1 is the lan interface

Main problem

The internal clients with ipv6 addresses from the /56 can't ping the outside world, but the packets and response is correctly routed up until my router, which just drops the packet without any hint.

e.g. ping6 ipv6.google.com results in timeouts.

but on the router's uplink I get:

20:43:13.350932 IP6 client-ipv6-ip > fra07s64-in-x00.1e100.net: ICMP6, echo request, seq 1, length 64
20:43:13.355143 IP6 fra07s64-in-x00.1e100.net > client-ipv6-ip: ICMP6, echo reply, seq 1, length 64
20:43:14.350572 IP6 client-ipv6-ip > fra07s64-in-x00.1e100.net: ICMP6, echo request, seq 2, length 64
20:43:14.354609 IP6 fra07s64-in-x00.1e100.net > client-ipv6-ip: ICMP6, echo reply, seq 2, length 64
20:43:15.350630 IP6 client-ipv6-ip > fra07s64-in-x00.1e100.net: ICMP6, echo request, seq 3, length 64
20:43:15.355072 IP6 fra07s64-in-x00.1e100.net > client-ipv6-ip: ICMP6, echo reply, seq 3, length 64
20:43:16.350656 IP6 client-ipv6-ip > fra07s64-in-x00.1e100.net: ICMP6, echo request, seq 4, length 64
20:43:16.354748 IP6 fra07s64-in-x00.1e100.net > client-ipv6-ip: ICMP6, echo reply, seq 4, length 64

on the client only the echo request is visible.

What could be the problem? Are there any additional settings so that my router is accepting anything that's sent to it?

Up until now I only used NAT and never full blown routing on my Linux boxes.

Thanks for any pointers, I hope it's just a simple sysctl parameter to set ...

Update 1

I replaced the /56 subnet by "subnet-2" and the /64 by "subnet-1".

ip -6 a:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
61: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qlen 1000
    inet6 subnet-1::2/64 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::216:2eff:fe01:1/64 scope link 
       valid_lft forever preferred_lft forever
63: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qlen 1000
    inet6 subnet-2::2/56 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::216:3eff:fe01:1/64 scope link 
       valid_lft forever preferred_lft forever

ip -6 r:

unreachable ::/96 dev lo  metric 1024  error -101
unreachable ::ffff:0.0.0.0/96 dev lo  metric 1024  error -101
unreachable 2002:a00::/24 dev lo  metric 1024  error -101
unreachable 2002:7f00::/24 dev lo  metric 1024  error -101
unreachable 2002:a9fe::/32 dev lo  metric 1024  error -101
unreachable 2002:ac10::/28 dev lo  metric 1024  error -101
unreachable 2002:c0a8::/32 dev lo  metric 1024  error -101
unreachable 2002:e000::/19 dev lo  metric 1024  error -101
subnet-1::/64 dev eth0  proto kernel  metric 256 
subnet-2::/56 dev eth1  proto kernel  metric 256 
unreachable 3ffe:ffff::/32 dev lo  metric 1024  error -101
fe80::/64 dev eth0  proto kernel  metric 256 
fe80::/64 dev eth1  proto kernel  metric 256 
default via fe80::1 dev eth0  metric 1024

System is up2date, except for the kernel (e.g. it's not loaded) since that will require a restart and I haven't had issues with the current one. But if it's a known issue with 3.10.0-123.13.2.el7 I can restart to upgrade it to 3.10.0-229.14.1.el7.

Update 2

tcpdump -i eth0 ip6

shows incoming packets.

I added a firewall rule to mangle e.g.

*mangle
-A PREROUTING -j NFLOG

to log to /var/log/messages, but nothing is logged there!

Update 3

    Bridge "br0"
        Port "router.eth0"
            Interface "router.eth0"
        Port "br0"
            Interface "br0"
                type: internal
        Port "eth0"
            Interface "eth0"
    Bridge "br1"
        Port "db01.eth0"
            Interface "db01.eth0"
        Port "mail01.eth0"
            Interface "mail01.eth0"
        Port "br1"
            Interface "br1"
                type: internal
        Port "team1"
            Interface "eth3"
            Interface "eth2"
        Port "router.eth1"
            Interface "router.eth1"

Update 4

I fucked up, had the wrong MAC-address on the interface (long story short - fixed it now).

So now the packets are accepted and forwarded, but the response is not sent to the default gateway:

Router:

22:09:22.638405 IP6 ping_ip > client_ip: ICMP6, echo request, seq 243, length 64
22:09:22.754936 IP6 router-ll > ff02::1:ff00:1: ICMP6, neighbor solicitation, who has fe80::1, length 32
22:09:22.757001 IP6 2a01:4f8::a:b:10 > router-ll: ICMP6, neighbor advertisement, tgt is fe80::1, length 32
22:09:22.757044 IP6 router-ll > ff02::1:ff00:2: ICMP6, neighbor solicitation, who has router-ll, length 32
22:09:23.639651 IP6 ping_ip > client_ip: ICMP6, echo request, seq 244, length 64
22:09:23.756969 IP6 router-ll > ff02::1:ff00:1: ICMP6, neighbor solicitation, who has fe80::1, length 32
22:09:23.759007 IP6 router-ll > ff02::1:ff00:2: ICMP6, neighbor solicitation, who has router-ll, length 32
22:09:23.759240 IP6 2a01:4f8::a:b:10 > router-ll: ICMP6, neighbor advertisement, tgt is fe80::1, length 32
22:09:24.640677 IP6 ping_ip > client_ip: ICMP6, echo request, seq 245, length 64

Client:

22:11:41.640978 IP6 ping_ip > client_ip: ICMP6, echo request, seq 382, length 64
22:11:41.640998 IP6 client_ip > ping_ip: ICMP6, echo reply, seq 382, length 64
22:11:42.642993 IP6 ping_ip > client_ip: ICMP6, echo request, seq 383, length 64
22:11:42.643013 IP6 client_ip > ping_ip: ICMP6, echo reply, seq 383, length 64

ip -6 neigh:

fe80::216:3eff:fe01:1 dev eth1 lladdr 00:16:3e:01:00:01 router STALE
subnet-2::1:1 dev eth0  FAILED
fe80::216:3eff:fe15:1 dev eth1 lladdr 00:16:3e:15:00:01 DELAY
fe80::216:3eff:fe15:c dev eth1 lladdr 00:16:3e:15:00:0c STALE
fe80::1 dev eth0  FAILED
subnet-2::1:1 dev eth1 lladdr 00:16:3e:15:00:01 REACHABLE
fe80::216:2eff:fe01:1 dev eth0  INCOMPLETE
subnet-2::1:12 dev eth1 lladdr 00:16:3e:15:00:0c STALE
subnet-1::2 dev eth0  FAILED

the failed entry for 1:1 is here because I shortly had that address on eth0 to debug the ping issue from before ...

Update 5

So as it happens with changing the MAC-address I didn't change the LL Address. I wasn't aware that those 2 addresses are linked 100%.

The main problem was that the old router died and I copied the LL Address from there didn't change the corresponding MAC-address → it didn't work.

Then ... changed the LL back (which was the mistake) and changed the MAC-address (in 2 separate steps a few hours apart).

For anyone else the L2 MAC und the IPv6 LL Address need to be corresponding, and they need to match with what a router is sending.

To debug use tcpdump -i eth0 ip6 -en then you can see the MAC-address and LL addresses and compare them with your interface, if any of them, and the one configured at Hetzner as routing address don't match, nothing works!

So again, sorry for anyone who read that, I somehow missed to explain that it was on a different router before and I installed this one because the last one died. Obvious fix ...

That's odd, you seem to be routing your /56 to your eth1 interface. Exactly how have you set up networking for your containers? — Michael Hampton, Oct 29 '15 at 21:40
I added the ovs-vsctl show output, the /56 should be routed in the internal network, each vm get's a static ip via configuration. I first had just the /56, Hetzner then suggested to add the /64 to make routing possible, that didn't work so we switched to LL routing — Thomas Rosenstein, Oct 29 '15 at 22:00

score 3 · Accepted Answer · answered Oct 30 '15 at 07:34

If you are experiencing weird routing issues with packets missing, follow these steps:

ip l and note the MAC Addresses

tcpdump -en -i eth0 ip6 (or the corresponding interface) and compare them, if they match all is good

ip -6 a compare the Link Local Addresses (scope local) to your mac address (here is a calculator: http://ben.akrin.com/?p=1347 , there are a few)

if it's still not working I guess the problem lies in the iptables, use logging as much as possible :)