Using libvirt within Docker container

Question

I am looking to perform a simple test on my VM and see how it behaves within a container and I want to test this using libvirt and QEMU/KVM. I have a domain .xml file that runs the VM just fine on a bare metal server but I am running into issues using libvirt within the container.

My Dockerfile looks like this:

FROM ubuntu:18.04

RUN apt-get update -qy \
 && apt-get upgrade -qy \
 && apt-get install -y \
    bridge-utils \
    iproute2 \
    python3-ipy \
    socat \
    qemu-kvm \
    libvirt-daemon-system \
    libvirt-clients \
    virtinst \
    bridge-utils \
    iptables \
    iproute2 \
    dnsmasq \
    vim \
 && rm -rf /var/lib/apt/lists/*

COPY *.qcow2 /
COPY *.xml /

I successfully create a simple image from this Dockerfile running sudo docker build . -t foo and to run the container I use sudo docker run -i --privileged -t --rm foo /bin/bash. This executes fine but once I am inserted into my containers shell, libvirt is giving me some issues. I am using a confirmed working libvirt domain XML file but when going to define it I get these errors in virsh:

error: failed to connect to the hypervisor
error: Failed to connect socket to '/var/run/libvirt/libvirt-sock': No such file or directory

On a normal environment not nested or containerized it's easier to debug issues with libvirt, usually the daemon is not enabled or running, in the case of a container I am unsure of how to exactly fix the issue. When I check the status of libvirt's daemon, I get this message

System has not been booted with systemd as init system (PID 1). Can't operate.

I assume I should be passing some arguments when running the container that has to do with my network?

There is no service manager (like systemd) running inside the container. If you want to start libvirtd you need to start it manually (e.g, via the `CMD` statement in a Dockerfile), and when it exits your container would exit. — larsks, Jul 13 '23 at 23:24
...but a Docker container is itself an isolated environment (and if you're using Docker Desktop or similar tools it is actually inside a virtual machine); it might not make sense to run a virtualization framework there. This tool might need to run directly on the host. — David Maze, Jul 14 '23 at 00:12
It's not as crazy as you might think; there are a number of reasons you might *want* to run libvirt inside a container. The [kubevirt](https://kubevirt.io) project does this, for example, and it allows your vm to access the same network environment that is visible to containerized workloads. — larsks, Jul 14 '23 at 02:35

score 0 · Answer 1 · answered Jul 14 '23 at 03:25

Following up on my comment, here's a basic Dockerfile for running a containerized libvirt:

FROM ubuntu:23.04

RUN export DEBIAN_FRONTEND=noninteractive && \
    apt-get update && \
    apt-get -y install \
        bridge-utils \
        dmidecode \
        dnsmasq \
        ebtables \
        iproute2 \
        iptables \
        libvirt-clients \
        libvirt-daemon-system \
        ovmf \
        qemu-efi \
        qemu-kvm \
        tini \
        && \
    apt-get clean

RUN sed -i '/^#stdio_handler/ a\stdio_handler = "file"' /etc/libvirt/qemu.conf

COPY config/pools/* /etc/libvirt/storage/
COPY config/networks/* /etc/libvirt/qemu/networks/
RUN mkdir -p /etc/libvirt/storage/autostart /etc/libvirt/qemu/networks/autostart && \
    for pool in /etc/libvirt/storage/*.xml; do \
        ln -sf "../${pool##*/}" /etc/libvirt/storage/autostart/; \
    done && \
    for net in /etc/libvirt/qemu/networks/*.xml; do \
        ln -sf "../${net##*/}" /etc/libvirt/qemu/networks/autostart/; \
    done

CMD ["/usr/bin/tini", "/usr/sbin/libvirtd"]

In config/pools/default.xml I have:

<pool type='dir'>
  <name>default</name>
  <target>
    <path>/var/lib/libvirt/images</path>
  </target>
</pool>

And in config/networks/default.xml I have:

<network>
  <name>default</name>
  <forward mode='nat'>
    <nat>
      <port start='1024' end='65535'/>
    </nat>
  </forward>
  <bridge name='virbr0' stp='on' delay='0'/>
  <ip address='192.168.122.1' netmask='255.255.255.0'>
    <dhcp>
      <range start='192.168.122.10' end='192.168.122.200'/>
    </dhcp>
  </ip>
</network>

If I build an image libvirtd from this Dockerfile and run it like this:

docker run -it --rm \
  -v /tmp/libvirt:/run/libvirt \
  --privileged \
  --name libvirt \
  --device /dev/kvm \
  -v /sys/fs/cgroup:/sys/fs/cgroup \
  libvirtd

Then I can do things like this:

$ virsh -c qemu:///system'?socket=/tmp/libvirt/libvirt-sock' net-list
 Name      State    Autostart   Persistent
--------------------------------------------
 default   active   yes         yes

Or even:

export LIBVIRT_DEFAULT_URI='qemu:///system?socket=/tmp/libvirt/libvirt-sock'

virsh vol-create-as --pool default jammy 653M
virsh vol-upload jammy \
  jammy-server-cloudimg-amd64.img \
  --pool default
virt-install -n jammy -r 4096 \
  --disk pool=default,size=10,backing_store=jammy,backing_format=qcow2 \
  --network network=default \
  --os-variant ubuntujammy \
  --import

Using libvirt within Docker container

1 Answers1