No IPv6 traffic in bridge mode

Question

Hello happy people o/~

yesterday I had the audacity to reboot a remote Linux server. This Ubuntu 64, 12.04 installation went from kernel 3.11.0-19-generic to 3.13.0-32-generic and also from being able to talk over both, IPv4 and IPv6 to being able to only receive and send IPv4 traffic.

### Hetzner's default network, modified to fit a bridge
# Loopback device:
auto lo
iface lo inet loopback

# device: eth0
auto  virbr1
iface virbr1 inet static
  address   5.9.87.134
  broadcast 5.9.87.159
  netmask   255.255.255.224
  gateway   5.9.87.129
  bridge_ports eth0
  bridge_stp on
  bridge_maxwait 0
  # Set-up IPv6 and Hetzner routes
  # guarantee idempotency:
  pre-up ip addr del 2a01:4f8:162:11c5::2/64 dev virbr1 || true
  up ip addr add 2a01:4f8:162:11c5::2/64 dev virbr1
  # default route to access subnet
  # idempotency here is easier, because we have 'replace'
  up ip route replace to 5.9.87.128/255.255.255.224 via 5.9.87.129 dev virbr1
  # add Hetzner IPv6 route
  up ip route replace default via fe80::1 dev virbr1

The result looks as follows:

1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr1 state UP qlen 1000
    link/ether c8:60:00:df:06:35 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::ca60:ff:fedf:635/64 scope link 
       valid_lft forever preferred_lft forever
3: virbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP 
    link/ether c8:60:00:df:06:35 brd ff:ff:ff:ff:ff:ff
    inet 5.9.87.134/27 brd 5.9.87.159 scope global virbr1
       valid_lft forever preferred_lft forever
    inet6 2a01:4f8:162:11c5::2/64 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::ca60:ff:fedf:635/64 scope link 
       valid_lft forever preferred_lft forever
4: virbr0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP 
    link/ether 52:54:00:65:05:b0 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
    inet6 2a01:4f8:162:11c5::10:1/112 scope global 
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe65:5b0/64 scope link 
       valid_lft forever preferred_lft forever
5: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc noop master virbr0 state DOWN qlen 500
    link/ether 52:54:00:65:05:b0 brd ff:ff:ff:ff:ff:ff
6: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast master virbr0 state UNKNOWN qlen 500
    link/ether fe:54:00:dc:99:21 brd ff:ff:ff:ff:ff:ff
    inet6 fe80::fc54:ff:fedc:9921/64 scope link 
       valid_lft forever preferred_lft forever

routing:

igalic@steak ~ % ip r
default via 5.9.87.129 dev virbr1  metric 100 
5.9.87.128/27 via 5.9.87.129 dev virbr1 
192.168.122.0/24 dev virbr0  proto kernel  scope link  src 192.168.122.1 
igalic@steak ~ % ip -6 r
2a01:4f8:162:11c5::10:0/112 dev virbr0  proto kernel  metric 256 
2a01:4f8:162:11c5::/64 dev virbr1  proto kernel  metric 256 
fe80::/64 dev eth0  proto kernel  metric 256 
fe80::/64 dev vnet0  proto kernel  metric 256 
fe80::/64 dev vnet1  proto kernel  metric 256 
fe80::/64 dev vnet2  proto kernel  metric 256 
fe80::/64 dev vnet3  proto kernel  metric 256 
fe80::/64 dev vnet4  proto kernel  metric 256 
fe80::/64 dev vnet5  proto kernel  metric 256 
fe80::/64 dev vnet6  proto kernel  metric 256 
fe80::/64 dev vnet7  proto kernel  metric 256 
fe80::/64 dev vnet8  proto kernel  metric 256 
fe80::/64 dev virbr0  proto kernel  metric 256 
fe80::/64 dev vnet9  proto kernel  metric 256 
fe80::/64 dev vnet10  proto kernel  metric 256 
fe80::/64 dev virbr1  proto kernel  metric 256 
default via fe80::1 dev virbr1  metric 1024 
igalic@steak ~ %

cut for brevity, because the server has a lot of VMs. It can communicate just fine with those VMs, over both, IPv4 and IPv6. It can communicate with the world just fine with IPv4 -- and so can its VMs.

Let's cut off here, and consider the question often asked from ##networking and comments here:

Why is your design so weird?

I use virbr1 as the external bridge for IPv4 addresses. By necessity of Hetzner's network design, or their recommendation, I have to bind the primary IPv6 address onto it.

see, http://wiki.hetzner.de/index.php/Zusaetzliche_IP-Adressen#IPv6_Subnetz (n.b.: this piece of recommendation on how to split the /64 doesn't exist in the English version of the same document: http://wiki.hetzner.de/index.php/Zusaetzliche_IP-Adressen/en )

But I only use the IPv6 addresses for Internal communication, that is, I use it only to connect services between my servers that don't need contact to the outside world: puppet, databases, etc… I cannot use ULA addresses here for the same reason I couldn't just stick with IPv4: The servers that need to communicate with each other are not in the same DCs, private IPv4 or ULAs would be useless.

I also need IPv4 on those internal links: I use them for provisioning. Hetzner won't allow PXE that's not controlled by them, so I have to run on setup server per server. (Have you ever tried preseed-ing over VPN? PXEbooting over IPv6? That's why)

After this short break, let's get back to…

IPv6: nothing works

The host cannot speak to the outside world via IPv6. The outside world cannot speak to the host via IPv6. The host can connect to its own virtual machines via IPv6. Virtual Machines can connect to the host via IPv6. But the virtual machines cannot connect to each other.

igalic@steak ~ % ping6 google.com -c5
PING google.com(fra07s30-in-x08.1e100.net) 56 data bytes
From steak icmp_seq=1 Destination unreachable: Address unreachable
From steak icmp_seq=2 Destination unreachable: Address unreachable
From steak icmp_seq=3 Destination unreachable: Address unreachable
From steak icmp_seq=4 Destination unreachable: Address unreachable
From steak icmp_seq=5 Destination unreachable: Address unreachable

--- google.com ping statistics ---
5 packets transmitted, 0 received, +5 errors, 100% packet loss, time 4024ms

1 igalic@steak ~ %

I have a second server with the exact setup which works. here's a ping from bacon (works) to steak (doesn't work):

igalic@bacon ~ % ping6 a01:4f:16:c5::2 -c5
PING 2a01:4f8:162:11c5::2(a01:4f:16:c5::2) 56 data bytes

--- a01:4f:16:c5::2 ping statistics ---
5 packets transmitted, 0 received, 100% packet loss, time 4032ms

1 igalic@bacon ~ %

and a tcpdump of what happens when I do that:

igalic@steak ~ % sudo tcpdump -vi any icmp6
[sudo] password for igalic: 
tcpdump: listening on any, link-type LINUX_SLL (Linux cooked), capture size 65535 bytes
10:17:31.712316 IP6 (hlim 59, next-header ICMPv6 (58) payload length: 64) 2a01:4f8:150:5024::2 > steak: [icmp6 sum ok] ICMP6, echo request, seq 1
10:17:31.712316 IP6 (hlim 59, next-header ICMPv6 (58) payload length: 64) 2a01:4f8:150:5024::2 > steak: [icmp6 sum ok] ICMP6, echo request, seq 1
10:17:31.712429 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 32) steak > ff02::1:ff00:1: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has fe80::1
          source link-address option (1), length 8 (1): c8:60:00:df:06:35
10:17:32.709170 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 32) steak > ff02::1:ff00:1: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has fe80::1
          source link-address option (1), length 8 (1): c8:60:00:df:06:35
10:17:32.718647 IP6 (hlim 59, next-header ICMPv6 (58) payload length: 64) 2a01:4f8:150:5024::2 > steak: [icmp6 sum ok] ICMP6, echo request, seq 2
10:17:32.718647 IP6 (hlim 59, next-header ICMPv6 (58) payload length: 64) 2a01:4f8:150:5024::2 > steak: [icmp6 sum ok] ICMP6, echo request, seq 2
10:17:33.709162 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 32) steak > ff02::1:ff00:1: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has fe80::1
          source link-address option (1), length 8 (1): c8:60:00:df:06:35
10:17:33.729483 IP6 (hlim 59, next-header ICMPv6 (58) payload length: 64) 2a01:4f8:150:5024::2 > steak: [icmp6 sum ok] ICMP6, echo request, seq 3
10:17:33.729483 IP6 (hlim 59, next-header ICMPv6 (58) payload length: 64) 2a01:4f8:150:5024::2 > steak: [icmp6 sum ok] ICMP6, echo request, seq 3
10:17:34.709180 IP6 (hlim 64, next-header ICMPv6 (58) payload length: 112) steak > steak: [icmp6 sum ok] ICMP6, destination unreachable, unreachable address 2a01:4f8:150:5024::2
10:17:34.709214 IP6 (hlim 64, next-header ICMPv6 (58) payload length: 112) steak > steak: [icmp6 sum ok] ICMP6, destination unreachable, unreachable address 2a01:4f8:150:5024::2
10:17:34.709239 IP6 (hlim 64, next-header ICMPv6 (58) payload length: 112) steak > steak: [icmp6 sum ok] ICMP6, destination unreachable, unreachable address 2a01:4f8:150:5024::2
10:17:34.735945 IP6 (hlim 59, next-header ICMPv6 (58) payload length: 64) 2a01:4f8:150:5024::2 > steak: [icmp6 sum ok] ICMP6, echo request, seq 4
10:17:34.735945 IP6 (hlim 59, next-header ICMPv6 (58) payload length: 64) 2a01:4f8:150:5024::2 > steak: [icmp6 sum ok] ICMP6, echo request, seq 4
10:17:34.736042 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 32) steak > ff02::1:ff00:1: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has fe80::1
          source link-address option (1), length 8 (1): c8:60:00:df:06:35
10:17:35.733165 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 32) steak > ff02::1:ff00:1: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has fe80::1
          source link-address option (1), length 8 (1): c8:60:00:df:06:35
10:17:35.745519 IP6 (hlim 59, next-header ICMPv6 (58) payload length: 64) 2a01:4f8:150:5024::2 > steak: [icmp6 sum ok] ICMP6, echo request, seq 5
10:17:35.745519 IP6 (hlim 59, next-header ICMPv6 (58) payload length: 64) 2a01:4f8:150:5024::2 > steak: [icmp6 sum ok] ICMP6, echo request, seq 5
10:17:36.733160 IP6 (hlim 255, next-header ICMPv6 (58) payload length: 32) steak > ff02::1:ff00:1: [icmp6 sum ok] ICMP6, neighbor solicitation, length 32, who has fe80::1
          source link-address option (1), length 8 (1): c8:60:00:df:06:35
10:17:37.733188 IP6 (hlim 64, next-header ICMPv6 (58) payload length: 112) steak > steak: [icmp6 sum ok] ICMP6, destination unreachable, unreachable address 2a01:4f8:150:5024::2
10:17:37.733215 IP6 (hlim 64, next-header ICMPv6 (58) payload length: 112) steak > steak: [icmp6 sum ok] ICMP6, destination unreachable, unreachable address 2a01:4f8:150:5024::2
^C
21 packets captured
26 packets received by filter
0 packets dropped by kernel
igalic@steak ~ %

disabling iptables has no effect on the behaviour.

This is all information I can imagine to be useful. I'm at a bit of a loss at this point, and I have no idea how to proceed with the debugging.

If you've copy-and-paste all those addresses (and I can't imagine you've typed them by hand), you're probably missing a `2` in a v6 address config file: `a01:4f:16:c5::2/64` is **not** a valid v6 address. If on the other hand you've deliberately munged the v6 address, that wasn't a particularly good way of doing it. — MadHatter, Aug 07 '14 at 09:57
Yes, the addresses are munged (*"changed to protect the innocent"*). I guess the fact that I'm doing it this level of wrong is another display of how much I know about IPv6. — Mina Galić, Aug 07 '14 at 10:05
Have you tried booting the older kernel again to see if the kernel version actually makes a difference? It may also be that your config scripts simply doesn't boot into the exact same network configuration, which you had before the first reboot. — kasperd, Aug 07 '14 at 11:34
@kasperd: tested right now, no difference. :( Please note that I regularly used to restart the network on these servers, and that doing that now doesn't "fix" it. — Mina Galić, Aug 07 '14 at 15:06
I find your description very confusing. what I deduce is that the server is connected to hetzner through eth0, and that the guests' interfaces are bridged to eth0 (server). Correct? If so, I don't get why the default route is through virbr1. Is the other server configured the same way? — countermode, Aug 09 '14 at 20:58
@countermode: I've updated with a short section explaining the reasoning behind my design. — Mina Galić, Aug 11 '14 at 09:19
From Hetzner's page: "Durch das Routing des Subnetzes auf die Link-Local-Adresse ergeben sich viele Möglichkeiten zur weiteren Aufteilung des Subnetzes" - they seriously suggest to subnet a /64. I am frankly speechless due to such ignorance and stupidity, they inflict designs on their customers that are bound to break. Yes one can manually assign addresses without SLAAC, but there's no technical need for such lousy crutches. Well, apart from that I still cannot see how packets are supposed to be forwarded with that setup. Indeed, Hetzner says: `ip route add default via fe80::1 dev eth0` — countermode, Aug 11 '14 at 10:46
more updates: I've had virtualbox installed on this machine, too. removing virtualbox-4.3 (and all of its dependencies) makes no difference. :( — Mina Galić, Aug 18 '14 at 22:10
@IgorGalić Sounds like some aspect of the network configuration was not touched by restarting the network service, since it kept working after the network service was restarted and broke once the machine was rebooted. — kasperd, Aug 20 '14 at 11:59
@countermode My recommendation in such cases is to ask the provider for a routed prefix. The more people asking for a routed prefix, the more likely they are to provide one. There exist no good reason for not giving customers a routed prefix. They have enough IPv6 addresses to give a /49 to every customer with an IPv4 address. — kasperd, Aug 20 '14 at 12:09

No IPv6 traffic in bridge mode

Why is your design so weird?

IPv6: nothing works

0 Answers0

Linked