0

I'm trying to set up a system whereby I have some process on a set of hosts that are in one (shielded) cpuset, while everything else (i.e. all the VMs) are in another. The goal here is to have a hyperconverged system where the VMs run on some arbitrary set of of the CPU cores, while the storage process run on the other cores, with isolation. I've selected cpuset to achieve this task as it seems like the "easiest" way to do it.

However this does not seem to work properly. VMs can be moved when first setting up the cset shield, but starting VMs later results in an error like:

libvirt: Cgroup error : Invalid value '0-31' for 'cpuset.cpus': Invalid argument

Clearly it's trying to put the VM in the root cset which is not allowed. This is with a very default configuration, just specifying a number of cores in the VM without any concern for individual CPU IDs or any such tuning (which is a requirement for this setup to work; nodes might be asymmetrical, have different core counts, topologies, etc. while VMs can live migrate around them).

But in general, I'd like to know: can Libvirt/KVM be configured to use a specific cpuset by default, without reconfiguring the VMs in any way (no manual config CPU pinning shenanigans), and without any manual tweaking on a per-process basis?

Joshua Boniface
  • 346
  • 3
  • 14

1 Answers1

1

On systemd based hosts, all VMs are placed in a cgroup beneath /machine.slice so if you set CPU mask for /machine.slice it'll apply to all VMs. Similarly if you set CPUAffinity in /etc/systemd/system.conf, that can make all non-VM processes avoid the CPUs used for VMs.

DanielB
  • 1,618
  • 7
  • 7
  • Excellent, I had seen machine.slice but had to remove it (forcibly) to enable the default shielding. I'll experiment a bit with manual sets rather than removing that. – Joshua Boniface Nov 08 '22 at 15:07
  • Marking this as accepted - I had never noticed this part of the cset before since I wasn't using --recurse and I was only seeing the "0 tasks" in the machine.slice cset, but this is indeed doing exactly what I wanted. – Joshua Boniface Nov 08 '22 at 15:31
  • As a bit of a follow-up, do you happen to know *how* systemd is doing that? Doing the same thing for my other processes seems optimal but I'm not sure what magic it's working! – Joshua Boniface Nov 08 '22 at 18:48
  • machine.slice is just a facade onto the correspondingly named cgroups, while I believe the systemd.conf CPUAffinity setting gets used to set the cpuset affinit y on the other top level .slice cgroups when they don't have a dedicated affinity set – DanielB Nov 09 '22 at 16:02
  • Thanks! I don't see any CPUAffinity settings set, and ended up posting this as its own question at https://serverfault.com/questions/1115179/assign-systemd-slice-to-a-specific-cset-automatically with a bit of my testing around it. – Joshua Boniface Nov 10 '22 at 06:51
  • See 'CPUAffinity' docs in 'man systemd-system.conf' – DanielB Nov 11 '22 at 12:38