1

I'm using systemd-networkd to configure network interfaces managed by libvirt for KVM (Kernel-based Virtual Machine) with Debian Bullseye on all nodes. I want to have transparent VLAN support on the virtual machines using the Linux Bridge. For the Linux Bridge this isn't supported by libvirt.

For example I have a virtual machine with three interfaces attached to the bridge with:

host ~$ virsh attach-interface guest-vm bridge br0 --config
host ~$ virsh attach-interface guest-vm bridge br0 --config
host ~$ virsh attach-interface guest-vm bridge br0 --config

Now when running the guest I will see on the bridge:

host ~$ sudo bridge link
3: enp1s0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br0 state forwarding priority 32 cost 4
30: vnet13: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br0 state forwarding priority 32 cost 100
31: vnet14: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br0 state forwarding priority 32 cost 100
32: vnet15: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br0 state forwarding priority 32 cost 100

All interfaces are successfully attached to the bridge. enp1s0 is the uplink interface on the host. Now when looking at the VLAN ids on the bridge I see:

~$ sudo bridge vlan
port              vlan-id
enp1s0            10
                  26
                  30
                  50

Only the host interface is shown with its VLAN ids.

Is there a way to also attach VLAN ids to the other interfaces vnet* of the guest so that it can use them?

Ingo
  • 416
  • 5
  • 13

3 Answers3

2

How to connect a libvirt vm to another network through a tagged vlan bridge:

Here's a similar setup for systems based on Redhat/CentOS using systemd-networkd, without shell scripts. This shows how to connect a libvirt/KVM vm to a network, the guest will receive a dynamic ip address automatically. The server will not be assigned an ip address on this network, so that no other computer within the network can see the server (for example, it could be an untrusted network for a Windows vm to communicate with a network scanner, printer or some device that requires you to use a buggy piece of software that only runs on Windows).

In this example, the server has a physical interface called "lan3", which is connected to a switch port which is a (tagged) member of several vlans. Each vlan has its own subnet and a dhcp server, for example: "lab1" vid 1910, subnet 192.168.10.0/24; "lab3" vid 1903, subnet 192.168.3.0/24 ...

Goal:

  • Server nic "lan3" = uplink
  • Server using systemd-networkd.
  • lan3 without vlan tag does not reach any network with dhcp, so no dynamic address assigned
  • Server uses "lan1" or some other nic for everything else (not relevant).
  • Upstream router or L3 switch provides several networks, one of them is "lab3".
  • Switch port provides these networks (tagged member).
  • Server must not receive an ip on this additional network.
  • Guest vm running Windows should receive an ip on this "lab3" network.
  • No manual interaction required, no scripting, it should work as long as systemd-networkd works.

Connections:

lan3 (phy) -> br3 (neutral) -> eth1930 (vlan) -> br_lab3 (vm bridge) -> vm

NIC config - uplink:

# lan3.network

[Match]
Name=lan3

[Network]
Bridge=br3

NIC config - uplink bridge device:

# br3.netdev 

[NetDev]
Name=br3
Kind=bridge

NIC config - uplink bridge network with VLAN membership(s):

# br3.network

[Match]
Name=br3

[Network]
VLAN=eth.1930

NIC config - VLAN device:

# eth.1930.netdev 

[NetDev]
Name=eth.1930
Kind=vlan

[VLAN]
Id=1930

NIC config - VLAN network:

# eth.1930.network 

[Match]
Name=eth.1930

[Network]
Bridge=br_lab3

# No IP for this server
[Network]
DHCP=no

NIC config - vm bridge device:

# br_lab3.netdev 

[NetDev]
Name=br_lab3
Kind=bridge

NIC config - vm bridge network:

# br_lab3.network 
[Match]
Name=br_lab3

[Network]
DHCP=no

VM config:

# bridge device = br_lab3

Explanation:

  • The uplink port "lan3" is not tagged, no ip assigned.
  • The uplink bridge "br3" is connected to the physical uplink port on one end and to the vlan with the generic name "eth.1930" on the other end. This vlan will allow us to join the "lab3" network. Other vlans could be added here...
  • This vlan is then connected to the vm bridge "br_lab3".
  • The vm bridge "br_lab3" is connected to the vm (network source = bridge device).
  • The guest system running inside that vm will receive a dynamic ip address from the uplink router that provides the network.

VM NIC config (vm bridge) in virt-manager

basic6
  • 353
  • 3
  • 9
1

I've worked on it and found a solution. it is no problem to manual add VLAN ids to the slave interfaces of a bridge, for example:

host ~$ sudo bridge vlan add vid 26 dev vnet13 pvid 26 untagged
host ~$ sudo bridge vlan add vid 30 dev vnet13
host ~$ sudo bridge vlan add vid 50 dev vnet14

host ~$ $ sudo bridge vlan
port              vlan-id
enp1s0            10
                  26
                  30
                  50
vnet13            26 PVID Egress Untagged
                  30
vnet14            50

The problem is to do it automatically on startup of the virtual machine. Fortunately libvirt supports libvirt hook scripts. I will use the hook script for qemu and do this in three steps.

Step 1: define which VLAN-ID is attached to what interface

For this we have an extra element <metadata> in Domain XML format for custom metadata. We can simply add the information to the static configuration of a domain (virtual machine):

host ~$ virsh edit guest-vm
--- snip ---
<metadata>
  <my:home xmlns:my="http://hoeft-online.de/libvirt">
    <my:iface pvid="26">
      <my:vlan untagged="yes">26</my:vlan>
      <my:vlan>50</my:vlan>
      <my:vlan untagged="no">30</my:vlan>
    </my:iface>
    <my:iface>
      <my:vlan untagged="yes">50</my:vlan>
      <my:vlan>10</my:vlan>
    </my:iface>
  </my:home>
</metadata>
--- snap ---

As given by the documentation I have to use my own custom namespace <my:home xmlns:my="http://hoeft-online.de/libvirt">. Once created it is easier to work within the namespace:

host ~$ virsh metadata guest-vm http://hoeft-online.de/libvirt [--edit --key my]
<home>
  <iface pvid="26">
    <vlan untagged="yes">26</vlan>
    <vlan>50</vlan>
    <vlan untagged="no">30</vlan>
  </iface>
  <iface>
    <vlan untagged="yes">50</vlan>
    <vlan>10</vlan>
  </iface>
</home>

Step 2: get information on startup from the runtime XML-config of the domain

The hook script gets information on its standard input. This is the XML-config of the running VM. We can also get it with virsh dumpxml guest-vm when the VM is running. I use XSLT with xmlstarlet to get needed information with a xsl-stylesheet. I can test with:

host ~$ virsh dumpxml guest-vm | xmlstarlet transform /etc/libvirt/hooks/qemu.xsl | xmlstarlet format

Here is the stylesheet:

host ~$ cat /etc/libvirt/hooks/qemu.xsl
<?xml version="1.0" encoding="UTF-8"?>
<!-- This stylesheet processes the live XML configuration output from a virtual
     machine managed by libvirt. It transforms the custom metadata information
     together with attached interfaces and returns a normalized XML with VLAN
     ids attached to the interface. For further information look at the
     README file.
     Author: 2021-01-26 - Ingo Höft (Ingo@Hoeft-online.de)
     Licence: GPLv3 (https://www.gnu.org/licenses/gpl-3.0.en.html)
-->
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform"
     xmlns:my="http://hoeft-online.de/libvirt" exclude-result-prefixes="my">
  <xsl:output omit-xml-declaration="yes" indent="no"
       encoding="utf-8" media-type="text/xml"/>
  <xsl:strip-space elements="*"/>
  <xsl:template match="text()|@*"/>


  <xsl:template match="/domain">
    <meta>
      <xsl:apply-templates/>
    </meta>
  </xsl:template>


  <xsl:template match='*'>
    <xsl:for-each select='interface[@type="bridge"]/target'>
    <iface>
      <xsl:variable name="_index" select="position()" />

      <xsl:attribute name="pvid">
        <xsl:choose>
          <xsl:when test="/*/metadata/my:home/my:iface[$_index]/@pvid != ''">
            <xsl:value-of select="/*/metadata/my:home/my:iface[$_index]/@pvid"/>
          </xsl:when>
          <xsl:otherwise>
            <xsl:text>0</xsl:text>
          </xsl:otherwise>
        </xsl:choose>
      </xsl:attribute>

      <xsl:value-of select="@dev"/>

      <xsl:for-each select="/*/metadata/my:home/my:iface[$_index]/my:vlan">
        <vlan>
          <xsl:attribute name="untagged">
            <xsl:choose>
              <xsl:when test="@untagged='yes'">
                <xsl:text>yes</xsl:text>
              </xsl:when>
              <xsl:otherwise>
                <xsl:text>no</xsl:text>
              </xsl:otherwise>
            </xsl:choose>
          </xsl:attribute>

          <xsl:value-of select="."/>
        </vlan>
      </xsl:for-each>

    </iface>
    </xsl:for-each>
  </xsl:template>

<!-- vim: set sts=2 sw=2 et autoindent nowrap: -->
</xsl:stylesheet>

Step 3: set VLAN-ID to the dynamic virtual network interface vnet*

With information from the stylesheet we can now setup the network interface with a hook script. Make it executable.

harley$ cat /etc/libvirt/hooks/qemu
#!/usr/bin/bash
# /etc/libvirt/hooks/qemu
# Docs: https://www.libvirt.org/hooks.html

# Author: 2021-01-26 - Ingo Höft (Ingo@Hoeft-online.de)
# Licence: GPLv3 (https://www.gnu.org/licenses/gpl-3.0.en.html)

# If you save a modified hook script then do 'sudo systemctl restart libvirtd'.

# This script adds VLAN support to interfaces of libvirt guests on start up.
# For more details look at the README file.
# Most work is done with the powerful XML transformation of the XML
# configuration of the guest on stdin with qemu.xsl to get a normalized
# meta information into $META, for example like this
# (we need it to understand the script):
#
# <?xml version="1.0"?>
# <meta>
#   <iface pvid="26">vnet1
#     <vlan untagged="yes">26</vlan>
#     <vlan untagged="no">50</vlan>
#     <vlan untagged="no">30</vlan>
#   </iface>
#   <iface pvid="30">vnet2
#     <vlan untagged="yes">30</vlan>
#     <vlan untagged="no">50</vlan>
#   </iface>
#   <iface pvid="0">vnet3</iface>
# </meta>

# for DEBUG uncomment/comment next three lines
#exec 0< start-vdeb11-base02.xml  # for DEBUG: read testfile to stdin
#BRIDGE="/usr/bin/echo"
BRIDGE="/usr/sbin/bridge"
# and call it with ./qemu "dummy-vm" "start" "begin" "-"

XSLFILE="/etc/libvirt/hooks/qemu.xsl"
XMLPROG="/usr/bin/xmlstarlet"

#GUEST=$1       # name of guest being started
OPERATION=$2
SUB_OPERATION=$3
EXTRA_PARM=$4


#echo 'DEBUG: entering qemu.hook' >&2
case "$OPERATION" in
    prepare)
      ;;
    start)
        if [[ "$SUB_OPERATION" != "begin" ]] || [[ "$EXTRA_PARM" != "-" ]]; then
            echo "Error: Unhandled parameter \$3='$SUB_OPERATION' or \$4='$EXTRA_PARM' to $0 \$1 \$2 \$3 \$4" >&2
            exit 1
        fi
        if [[ ! -x "$XMLPROG" ]]; then
            echo "Error: $XMLPROG is not executable" >&2
            exit 1
        fi

        #cat - >/var/log/libvirt/start-"$1".xml; exit 1   # get live xml for DEBUG
        META=$("$XMLPROG" tr "$XSLFILE" -)
        #echo "DEBUG: using hook start with $META" >&2

        # loop through interfaces
        IFACE_COUNT=0
        while true; do
            ((++IFACE_COUNT))
            IFACE=$(echo "$META" | "$XMLPROG" sel -t -c "/meta/iface[$IFACE_COUNT]/text()")
            if [[ -z "$IFACE" ]]; then
                # finished, no more interfaces available
                exit 0
            fi

            "$BRIDGE" link set dev "$IFACE" flood off

            # loop through vlans on one interface
            VLAN_COUNT=0
            while true; do
                ((++VLAN_COUNT))
                VLAN=$(echo "$META" | "$XMLPROG" sel -t -v "/meta/iface[$IFACE_COUNT]/vlan[$VLAN_COUNT]/text()")
                if [[ -z "$VLAN" ]]; then
                    # finished, no more vlans available, process next interface
                    break
                else
                    UNTAGGED=$(echo "$META" | "$XMLPROG" sel -t -v "/meta/iface[$IFACE_COUNT]/vlan[$VLAN_COUNT]/@untagged")
                    if [[ "$UNTAGGED" == "yes" ]]; then
                        "$BRIDGE" vlan add vid "$VLAN" dev "$IFACE" pvid "$VLAN" untagged
                    else
                        "$BRIDGE" vlan add vid "$VLAN" dev "$IFACE"
                    fi
                fi
            done
        done
        ;;
    started)
        ;;
    stopped)
        ;;
    release)
        ;;
    migrate)
        ;;
    restore)
        ;;
    reconnect)
        ;;
    attach)
        ;;
    *)
        echo "Error: qemu hook called with unexpected options $*" >&2
        exit 1
        ;;
esac

# vim: set sts=4 sw=4 et autoindent nowrap:
Ingo
  • 416
  • 5
  • 13
0

Thank you @Ingo for your idea, while his (propably) solution works great, I didn't really want to use xmlstarlet as it is not in the repos of Centos Stream 9.

Anyways I used Ingos idea and implemented it in python. The only thing needed is python.

https://github.com/modzilla99/libvirt-vlans

modzilla
  • 9
  • 1