Is there a way to create a bridge using interfaces with static routes?

Question

For context, I'm setting up a 3-node Proxmox cluster for personal use and am setting up 2 different mesh networks on 4 interfaces (2 interfaces per node for each mesh network). That way each node has a direct connection to every other node. To get networking to work I'm depending on static routes on each interface, configured as such:

auto lo
iface lo inet loopback

iface enp5s0f0 inet manual

iface enp5s0f1 inet manual

auto eno1
iface eno1 inet static
        address 10.83.86.10/24
        up ip route add 10.83.86.11/32 dev eno1
        down ip route del 10.83.86.11/32

auto eno2
iface eno2 inet static
        address 10.83.86.10/24
        up ip route add 10.83.86.12/32 dev eno2
        down ip route del 10.83.86.12/32

auto eno3
iface eno3 inet static
        address 10.83.67.10/24
        up ip route add 10.83.67.11/32 dev eno3
        down ip route del 10.83.67.11/32

auto eno4
iface eno4 inet static
        address 10.83.67.10/24
        up ip route add 10.83.67.12/32 dev eno4
        down ip route del 10.83.67.12/32

auto vmbr0
iface vmbr0 inet static
        address 192.168.0.20/24
        gateway 192.168.0.1
        bridge-ports enp5s0f0
        bridge-stp off
        bridge-fd 0

source /etc/network/interfaces.d/*

IP addresses differ between nodes, but otherwise the configuration is the same. Interfaces eno1 and eno2 are part of the 10.83.86.0 network, and eno3 and eno4 are part of the 10.83.67.0 network.

The problem arises that I don't know how to expose these networks to VMs and containers running on Proxmox. For example, I want to be able to load-balance and proxy tunnel the web interface for Proxmox, which I'm planning to only expose on the mesh network. Another example is that I need certain workloads to be able to access the Ceph public network, such as the Ceph Kubernetes CSI driver.

As I understand it, Proxmox requires a Linux bridge for virtual network card virtualization. I've tried to rewrite my interface configuration as:

auto eno1
iface eno1 inet manual
        up ip route add 10.83.86.11/32 dev eno1
        down ip route del 10.83.86.11/32

auto eno2
iface eno2 inet manual
        up ip route add 10.83.86.12/32 dev eno2
        down ip route del 10.83.86.12/32

auto vmbr1
iface vmbr1 inet static
        address 10.83.86.10/24
        bridge-ports eno1 eno2
        bridge-stp off
        bridge-fd 0

But this configuration just... breaks networking between nodes when I reload with ifreload -a. I don't fully understand why.

Nikita Kipriyanov · Accepted Answer · 2022-12-26T19:51:53.627

Routes belong to IP interfaces, not bridge ports. Your bridge ports shouldn't contain literally anything. All routes are associated with interfaces that have IP addresses.

Consider if you had this:

auto eno1
iface eno1 inet static
        address 10.83.86.10/24
        up ip route add 10.83.86.11/32 dev eno1
        down ip route del 10.83.86.11/32

then it becomes this:

iface eno1 inet manual

auto vmbr1
iface vmbr1 inet static
        address 10.83.86.10/24
        up ip route add 10.83.86.11/32 dev vmbr1
        down ip route del 10.83.86.11/32
        bridge-ports eno1
        bridge-stp off
        bridge-fd 0

Notice how all IP-related stuff just simply moved from physical NIC into bridge. The NIC becomes L2-only interface, think of it as a switch port. Switches don't have IP addresses on each interface, neither bridged interfaces should.

You can bridge several physical NICs as you did by specifying them all in bridge-ports. None of the ports will have an IP configuration. Only the bridge can have it.

I can't understand what you are going to achieve by having each interface individual /32 address and individual route. I don't know why you want that, but this network setup looks rather strange.

Yet if you want to keep the network (layer 3) structure as you have in your question, you need to create a dedicated bridge for each IP address and put respective VM into that bridge.

If this all is just for host-to-VM communication, you don't need to bring any physical NIC into those bridges. Leave vmbrX without any bridge ports, but configure an IP addresses. You will be able to put VMs into that "empty" bridge and they will communicate between each other and the host. The physical NICs only need to be part of the bridge if you need to connect this virtual network segment with physical network on layer 2 (e.g. Ethernet).

Is there a way to create a bridge using interfaces with static routes?

1 Answers1