3

I'm running Linux under QEMU as part of a test suite for a Python script that does some privileged operations. Running a full virtual machine is important to me, because:

  1. I do not want to require sudo access to run tests
  2. later it will be easier to run the tests across different distributions and kernel versions
  3. I'm less likely to break my computer

My approach is to start the QEMU VM with a host port forwarded to SSH on the VM, send and run the script over SSH, run the script, and assert things about the remote system. The annoying thing about this setup is the SSH port, and ports for other network applications.

My question: How can I forward ports in user-mode QEMU without conflicts?

My goal here is to be able to run multiple tests concurrently without any prior setup and without sudo, as mentioned above. User-mode networking in QEMU supports port forwarding, and when I pass 0 for the host port in the hostfwd declaration (hostfwd=tcp:127.0.0.1:0-:22) the OS allocates one dynamically. Awesome! The problem with that has been getting that port reliably in the parent process.

I'm doing the equivalent of:

#!/bin/sh
set -e

cd "$(mktemp -d)"

# Download cloud image.
image_base_url="https://cloud-images.ubuntu.com/releases/18.04/release/"
image_name="ubuntu-18.04-server-cloudimg-amd64.img"
image_url="$image_base_url$image_name"
curl --location -o linux.img $image_url

# Create SSH key.
ssh_key_path="$PWD/id_rsa"
ssh-keygen -t rsa -b 4096 -N "" -f "$ssh_key_path"

# Create cloud-init payload.
cat <<EOF > user_data
#cloud-config
password: ubuntu
chpasswd: { expire: False }
ssh_pwauth: True
ssh_authorized_keys:
- $(cat "${ssh_key_path}.pub")
EOF
cloud-localds user_data.img user_data

# Socket path for QMP.
qmp_socket="$PWD/qmp.sock"

# Start VM.
qemu-system-x86_64 \
    -drive file=linux.img,format=qcow2 \
    -drive file=user_data.img,format=raw \
    -netdev user,id=net0,hostfwd=tcp:127.0.0.1:0-:22 \
    -device rtl8139,netdev=net0 \
    -enable-kvm \
    -m 2G \
    -serial mon:stdio \
    -nographic \
    -smp 2 \
    -snapshot \
    -qmp unix:$qmp_socket,server,nowait \
    & \
;

# wait a bit, then test stuff here
# ...

The guest OS comes up fine.

Things I have tried:

  • played a little with QMP, but couldn't find a command that provided the information

  • considered binding a random port in the parent process with SO_REUSEPORT and then specifying it for the child command, but QEMU doesn't use this flag, so it doesn't work

  • netstat -nalp | grep $child_pid, but this will not be usable when forwarding multiple ports

  • picking a random unbound port, trying to start qemu with it, and trying again if it fails with a message matching Could not set up host forwarding rule 'tcp:...'. This works, but

    1. is more likely to fail with more ports
    2. may obscure other issues that aren't worth retrying for

    It's an OK fallback, but I'm holding out hope for a more straightforward solution.

Chris Hunt
  • 171
  • 1
  • 8

1 Answers1

2

After a little more research: QEMU exposes the information via the info usernet command. Here's what the output looks like:

(qemu) info usernet
VLAN -1 (net0):
  Protocol[State]    FD  Source Address  Port   Dest. Address  Port RecvQ SendQ
  TCP[HOST_FORWARD]  13       127.0.0.1 38117       10.0.2.15    22     0     0
  UDP[236 sec]       24       10.0.2.15 35061   91.189.89.198   123     0     0
  UDP[204 sec]       26       10.0.2.15 60630   91.189.89.198   123     0     0

It's not available via QMP, but there was a patch submitted to add an equivalent query-usernet here some time ago.

In my case, it is straightforward to expose the monitor using -monitor unix:$PWD/mon.sock,server,nowait, connect, send the command, read the output, and parse it for the desired values.

So in summary, to use QEMU to test network applications without requiring sudo or other prereqs (other than QEMU itself), and avoiding port conflicts on the host:

  1. For each application, create a port mapping via the command-line (-netdev) or monitor (netdev_add), providing 0 for the host port
  2. Expose the monitor as a unix socket for the consuming application
  3. Execute info usernet via the socket, and extract the Source Port for each mapping, which is the actual port allocated by the OS
Chris Hunt
  • 171
  • 1
  • 8