for the past week or so i've been recovering from the shock that i have literally no idea how my isp's network works, by getting a new box and trying to do a sensible design from scratch.
our design constraints are as follows:
- libvirt + kvm (because we're poor)
- ubuntu (12.04 on hosts, but we're willing to try 14.04 if openvswitch were to make things easier)
- one external network interface (we can't change this)
- one ipv4, a /64 ipv6, and limited additional (public) ipv4 addresses, with an… uh… interesting setup
these constraints lead us to the following considerations:
- public ip addresses should be uses sparsely, we use them mostly for customer facing caching proxies
- all virtual machines on all servers must be able to talk to each other (proxy to app server. app server to database. all servers to mgmt infra. &c…)
- since we already have /64 IPv6 addresses, why not use those?!
this however leads us to several problems, because ipv6 support isn't quite perfect everywhere yet:
- post-pxe boot (from a temporary private IPv4 dhcp server) we still need to access several ipv4 resources during vm provisioning
- at this stage (in busybox) we cannot setup a 4over6 tunnel, because the tools don't support it
- but even afterwards we might still have to access some v4-only resources (apt repositories, github, &c…)
- a 4over6 tunnel seems hackish, at best.
but wait? what's wrong with the setup that you have?!
currently we have one bridge, directly exposed to libvirt for the public ipv4 addresses. and we have one virtual bridge with nat that serves for installation of those hidden VMs AND provides routing for our IPv6 addresses.
the problem with this setup is that it is hugely flakey: right now i have one box which doesn't respond to anything other than ping, but who's vms are still reachable (ipv4 and v6) and one box which is perfectly interconnected, but who's VMs cannot get out over ipv6.
perhaps this is because i'm finding hetzner's documentation impenetrable, or because i simply do not understand the deeper design issues at their hands, or because i don't know network.
i'm looking for a design that's… less unstable. even if that means i'll have to (re)learn network.
