Docker container on QEMU guest has no LAN-connection

Question

On a Ubuntu 20.04 server I run a Debian 10 as a KVM-guest, connected via a macvtap-bridge. Inside that guest I have a macvlan-bridge, connecting a docker container.

For some reason that container does not seem to have any network connection and I'm unable to see the error.

edit2

Since I didn't get any answer yet and since I also continued to investigate the whole topic, I'm getting some doubts whether my idea does actually work at all!

Therefore I would like to ask very generally upfront: Is it possible to have a KVM-guest, that runs docker with some container and have them all connected via a "public bridge"? So that the host, the KVM-guest and the docker container appear all on the local LAN just like separate machines (and of course can communicate with each other). This type of setup is referred to as "nested virtualization". But I was unable to find some references that explain how to "public bridge" both levels of nested VMs.

So would be nice if someone could clarify this question first, in the best of cases including a documentation/explanation on how to do it.

/edit2

edit1

To give a little more background, I basically try to achieve a setup like this:

   +-------------------------------+
   | +---+---+---+   +-----------+ |
   | |Cn1|Cn2|CnX|   |Cn1|Cn2|CnX| |
   | +---+---+---+   +---+---+---+ |
   | |  Docker   |   |  Docker   | |
   | +-----------+   +-----------+ |
   | |  macvlan  |   |  macvlan  | |
   | |VM1 eth0   |   |VM2 eth0   | |
   | +-----------+   +-----------+ |
   |    macvtap         macvtap    |
   |    +----+          +----+     |
   |    |eth0|          |eth1|     |
   |    +----+   Host   +----+     |
   +-------------------------------+
                  |
                 LAN

So the Host, VM1, VM2 and the docker container should be connected directly to the LAN just as they were physical machines. So if anyone can point me to a different approach to achieve this, I would be happy as well...

/edit1

Here are the configuration details:

Ubuntu host (banana)

/etc/network/interfaces:

# The primary network interface
auto eth0
iface eth0 inet static
        address 192.168.3.3
        netmask 255.255.255.0
        gateway 192.168.3.1
        dns-search *****.local
        dns-nameserver 192.168.3.1

# USB-ethernet adapter
auto eth1
iface eth1 inet static
        address 192.168.2.12
        netmask 255.255.255.0
        broadcast 255.255.255.255
        dns-nameservers 192.168.2.250
        dns-search *****.local

ip addr:

root@banana:~# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether dc:fe:07:e1:80:e6 brd ff:ff:ff:ff:ff:ff
    inet 192.168.3.3/24 brd 192.168.3.255 scope global eth0
       valid_lft forever preferred_lft forever
    inet6 fe80::b0d2:f660:df88:856f/64 scope link 
       valid_lft forever preferred_lft forever
3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether 00:e0:4c:68:13:97 brd ff:ff:ff:ff:ff:ff
    inet 192.168.2.12/24 brd 255.255.255.255 scope global eth1
       valid_lft forever preferred_lft forever
    inet 192.168.2.104/24 brd 192.168.2.255 scope global secondary noprefixroute eth1
       valid_lft forever preferred_lft forever
    inet6 fdc3:1fe4:1492::eb8/128 scope global noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 fdc3:1fe4:1492:0:942c:ded8:d9ee:2862/64 scope global temporary dynamic 
       valid_lft 600505sec preferred_lft 81850sec
    inet6 fdc3:1fe4:1492:0:1b46:b4f9:692c:2f10/64 scope global mngtmpaddr noprefixroute 
       valid_lft forever preferred_lft forever
    inet6 fe80::63b7:ce64:c7d1:73da/64 scope link 
       valid_lft forever preferred_lft forever
4: virbr1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 52:54:00:02:0c:0d brd ff:ff:ff:ff:ff:ff
    inet 192.168.254.1/24 brd 192.168.254.255 scope global virbr1
       valid_lft forever preferred_lft forever
    inet 169.254.70.129/16 brd 169.254.255.255 scope global noprefixroute virbr1
       valid_lft forever preferred_lft forever
5: virbr1-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel master virbr1 state DOWN group default qlen 1000
    link/ether 52:54:00:02:0c:0d brd ff:ff:ff:ff:ff:ff
6: virbr0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default qlen 1000
    link/ether 52:54:00:44:21:01 brd ff:ff:ff:ff:ff:ff
    inet 192.168.122.1/24 brd 192.168.122.255 scope global virbr0
       valid_lft forever preferred_lft forever
7: virbr0-nic: <BROADCAST,MULTICAST> mtu 1500 qdisc fq_codel master virbr0 state DOWN group default qlen 1000
    link/ether 52:54:00:44:21:01 brd ff:ff:ff:ff:ff:ff
8: macvtap0@eth1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 500
    link/ether 52:54:00:be:09:bc brd ff:ff:ff:ff:ff:ff
    inet 192.168.2.63/24 brd 192.168.2.255 scope global noprefixroute macvtap0
       valid_lft forever preferred_lft forever
    inet6 fe80::b4bb:1ac3:ab06:b569/64 scope link 
       valid_lft forever preferred_lft forever
9: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel master virbr1 state UNKNOWN group default qlen 1000
    link/ether fe:54:00:1f:e8:10 brd ff:ff:ff:ff:ff:ff
    inet 169.254.134.94/16 brd 169.254.255.255 scope global noprefixroute vnet0
       valid_lft forever preferred_lft forever
    inet6 fe80::ea3:8aaf:a558:b1b6/64 scope link 
       valid_lft forever preferred_lft forever
    inet6 fe80::fc54:ff:fe1f:e810/64 scope link 
       valid_lft forever preferred_lft forever

Libvirt XML-part of network for the Debian-guest:

<interface type='direct'>
  <mac address='52:54:00:be:09:bc'/>
  <source dev='eth1' mode='bridge'/>
  <model type='virtio'/>
  <address type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</interface>
<interface type='network'>
  <mac address='52:54:00:1f:e8:10'/>
  <source network='isolated'/>
  <model type='virtio'/>
  <address type='pci' domain='0x0000' bus='0x09' slot='0x00' function='0x0'/>
</interface>

Debian guest (bananaLAN)

/etc/network/interfaces:

# The primary network interface
auto enp1s0
iface enp1s0 inet static
        address 192.168.2.3
        netmask 255.255.255.0
        gateway 192.168.2.1

# for host communication
auto enp9s0
iface enp9s0 inet static
        address 192.168.254.3
        netmask 255.255.255.0

ip addr

root@bananaLAN:/# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
2: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:be:09:bc brd ff:ff:ff:ff:ff:ff
    inet 192.168.2.3/24 brd 192.168.2.255 scope global enp1s0
       valid_lft forever preferred_lft forever
    inet6 fdc3:1fe4:1492:0:5054:ff:febe:9bc/64 scope global dynamic mngtmpaddr 
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:febe:9bc/64 scope link 
       valid_lft forever preferred_lft forever
3: enp9s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UP group default qlen 1000
    link/ether 52:54:00:1f:e8:10 brd ff:ff:ff:ff:ff:ff
    inet 192.168.254.3/24 brd 192.168.254.255 scope global enp9s0
       valid_lft forever preferred_lft forever
    inet6 fe80::5054:ff:fe1f:e810/64 scope link 
       valid_lft forever preferred_lft forever
4: docker0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN group default 
    link/ether 02:42:0d:e8:3a:25 brd ff:ff:ff:ff:ff:ff
    inet 172.17.0.1/16 brd 172.17.255.255 scope global docker0
       valid_lft forever preferred_lft forever

Docker (routix)

network create

docker network create \
  --driver=macvlan \
  --subnet=192.168.2.0/24 \
  --gateway 192.168.2.1 \
  --ip-range=192.168.2.224/27 \
  -o parent=enp1s0 \
  pub_net

container create

docker run -d \
    --name routix \
    --network=pub_net \
    --ip 192.168.2.250 \
    -p 53:53/tcp -p 53:53/udp \
    -p 80:80 \
    -p 443:443 \
    -e TZ="Europe/Berlin" \
    -v "$(pwd)/etc-pihole/:/etc/pihole/" \
    -v "$(pwd)/etc-dnsmasq.d/:/etc/dnsmasq.d/" \
    --dns=127.0.0.1 --dns=192.168.3.1 \
    --restart=unless-stopped \
    pihole/pihole:latest

ip addr:

root@3c59f964891c:/# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
5: eth0@if2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default 
    link/ether 02:42:c0:a8:02:fa brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 192.168.2.250/24 brd 192.168.2.255 scope global eth0
       valid_lft forever preferred_lft forever

Ping results

Ubuntu host (banana) to gateway:

root@banana:~# ping 192.168.2.1
PING 192.168.2.1 (192.168.2.1) 56(84) Bytes Daten.
64 Bytes von 192.168.2.1: icmp_seq=1 ttl=64 Zeit=0.585 ms
64 Bytes von 192.168.2.1: icmp_seq=2 ttl=64 Zeit=0.572 ms

Ubuntu host (banana) to LAN-host:

root@banana:~# ping 192.168.2.20
PING 192.168.2.20 (192.168.2.20) 56(84) Bytes Daten.
64 Bytes von 192.168.2.20: icmp_seq=1 ttl=64 Zeit=0.810 ms
64 Bytes von 192.168.2.20: icmp_seq=2 ttl=64 Zeit=0.545 ms

Ubuntu host (banana) to Debian guest: (that's expected)

root@banana:~# ping 192.168.2.3
PING 192.168.2.3 (192.168.2.3) 56(84) Bytes Daten.
Von 192.168.2.12 icmp_seq=1 Zielhost nicht erreichbar
Von 192.168.2.12 icmp_seq=2 Zielhost nicht erreichbar

Ubuntu host (banana) to Debian guest via Isolated-network:

root@banana:~# ping 192.168.254.1
PING 192.168.254.1 (192.168.254.1) 56(84) Bytes Daten.
64 Bytes von 192.168.254.1: icmp_seq=1 ttl=64 Zeit=0.222 ms
64 Bytes von 192.168.254.1: icmp_seq=2 ttl=64 Zeit=0.065 ms

Debian guest (bananaLAN) to gateway:

root@bananaLAN:/# ping 192.168.2.1
PING 192.168.2.1 (192.168.2.1) 56(84) bytes of data.
64 bytes from 192.168.2.1: icmp_seq=1 ttl=64 time=0.709 ms
64 bytes from 192.168.2.1: icmp_seq=2 ttl=64 time=0.701 ms

Debian guest (bananaLAN) to LAN-host:

root@bananaLAN:/# ping 192.168.2.20
PING 192.168.2.20 (192.168.2.20) 56(84) bytes of data.
64 bytes from 192.168.2.20: icmp_seq=1 ttl=64 time=0.938 ms
64 bytes from 192.168.2.20: icmp_seq=2 ttl=64 time=0.872 ms

Debian guest (bananaLAN) to Ubuntu host: (that's expected)

root@bananaLAN:/# ping 192.168.2.12
PING 192.168.2.12 (192.168.2.12) 56(84) bytes of data.
From 192.168.2.3 icmp_seq=1 Destination Host Unreachable
From 192.168.2.3 icmp_seq=2 Destination Host Unreachable

Debian guest (bananaLAN) to Ubuntu host via Isolated-network:

root@bananaLAN:/# ping 192.168.254.3
PING 192.168.254.3 (192.168.254.3) 56(84) bytes of data.
64 bytes from 192.168.254.3: icmp_seq=1 ttl=64 time=0.098 ms
64 bytes from 192.168.254.3: icmp_seq=2 ttl=64 time=0.039 ms

Docker container (routix) to gateway: (that's the problem!)

root@bananaLAN:/# docker exec -it routix bash
root@3c59f964891c:/# ping 192.168.2.1
PING 192.168.2.1 (192.168.2.1) 56(84) bytes of data.
From 192.168.2.250 icmp_seq=1 Destination Host Unreachable
From 192.168.2.250 icmp_seq=2 Destination Host Unreachable

Docker container (routix) to LAN-host: (that's the problem!)

root@3c59f964891c:/# ping 192.168.2.20
PING 192.168.2.20 (192.168.2.20) 56(84) bytes of data.
From 192.168.2.250 icmp_seq=1 Destination Host Unreachable
From 192.168.2.250 icmp_seq=2 Destination Host Unreachable

Docker container (routix) to Ubuntu-host:

root@3c59f964891c:/# ping 192.168.2.12
PING 192.168.2.12 (192.168.2.12) 56(84) bytes of data.
From 192.168.2.250 icmp_seq=1 Destination Host Unreachable
From 192.168.2.250 icmp_seq=2 Destination Host Unreachable

Docker container (routix) to Debian-host:

root@3c59f964891c:/# ping 192.168.2.3 
PING 192.168.2.3 (192.168.2.3) 56(84) bytes of data.
From 192.168.2.250 icmp_seq=1 Destination Host Unreachable
From 192.168.2.250 icmp_seq=2 Destination Host Unreachable

LAN-host to gateway:

~ % ping 192.168.2.1
PING 192.168.2.1 (192.168.2.1): 56 data bytes
64 bytes from 192.168.2.1: icmp_seq=0 ttl=64 time=0.392 ms
64 bytes from 192.168.2.1: icmp_seq=1 ttl=64 time=0.406 ms

LAN-host to Ubuntu-host (banana):

~ % ping 192.168.2.12
PING 192.168.2.12 (192.168.2.12): 56 data bytes
64 bytes from 192.168.2.12: icmp_seq=0 ttl=64 time=0.593 ms
64 bytes from 192.168.2.12: icmp_seq=1 ttl=64 time=0.652 ms

LAN-host to Debian-host (bananaLAN):

~ % ping 192.168.2.3 
PING 192.168.2.3 (192.168.2.3): 56 data bytes
64 bytes from 192.168.2.3: icmp_seq=0 ttl=64 time=0.927 ms
64 bytes from 192.168.2.3: icmp_seq=1 ttl=64 time=1.025 ms

LAN-host to Docker-container (routix): (that's the problem!)

~ % ping 192.168.2.250
PING 192.168.2.250 (192.168.2.250): 56 data bytes
Request timeout for icmp_seq 0
Request timeout for icmp_seq 1

Routing tables

Gateway (192.168.2.1)

root@OpenWrt:~# ip route
default via 192.168.3.1 dev eth0.10 
192.168.2.0/24 dev br-lan scope link  src 192.168.2.1 
192.168.3.0/24 dev eth0.10 scope link  src 192.168.3.2 

Ubuntu host (banana)

root@banana:~# ip route
default via 192.168.3.1 dev eth0 src 192.168.3.3 metric 202 
default via 192.168.2.1 dev eth1 proto dhcp src 192.168.2.104 metric 203 
default via 192.168.2.1 dev macvtap0 proto dhcp src 192.168.2.63 metric 208 
169.254.0.0/16 dev virbr1 scope link src 169.254.70.129 metric 204 
169.254.0.0/16 dev vnet0 scope link src 169.254.134.94 metric 209 
192.168.2.0/24 dev eth1 proto dhcp scope link src 192.168.2.104 metric 203 
192.168.2.0/24 dev macvtap0 proto dhcp scope link src 192.168.2.63 metric 208 
192.168.3.0/24 dev eth0 proto dhcp scope link src 192.168.3.3 metric 202 
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 linkdown 
192.168.254.0/24 dev virbr1 proto kernel scope link src 192.168.254.1

Debian guest (bananaLAN)

root@bananaLAN:/# ip route
default via 192.168.2.1 dev enp1s0 onlink 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 
192.168.2.0/24 dev enp1s0 proto kernel scope link src 192.168.2.3 
192.168.254.0/24 dev enp9s0 proto kernel scope link src 192.168.254.3

Docker container (routix)

root@3c59f964891c:/# ip route
default via 192.168.2.1 dev eth0 
192.168.2.0/24 dev eth0 proto kernel scope link src 192.168.2.250

Probably this is some kind of routing problem and I have to admit, that this is not amongst my strengths. I also triple-checked to have followed exactly the several documentations around qemu/libvirt and docker/macvlan, but obviously I'm missing something.

I appreciate any help! Thanks a lot!