How to set the Linux init process affinity without using grub?

Question

Our application may run on a variety of architectures all on a ramfs machine. It needs to set core isolation and affinity for extremely fast processing. Depending on the architecture, it may need to be pinned to a hyperthread on numa 0 or numa 1. We cannot make assumptions until the machine boots.

Because of this variety, we cannot set isolation parameters in grub. Rebooting after configuring grub will not help either, as this is a ramfs machine.

How can we taskset all userland processes spawned by systemd during boot? I know I can taskset systemd itself when it is invoked, but I cannot find where that happens.

_Depending on the architecture_ - when does this determination happen? — Maxim Egorushkin, Jan 21 '19 at 11:01
As soon as we boot. It depends on the location of devices in the PCIe realm. So we need to first figure out if a device (say - a NIC) is in a PCI slot connected to one NUMA or another, or perhaps even both, and then determine our application threads layout. We even have systems that have 4 NUMA nodes, so we may need to spread ourselves over all of them to achieve optimal performance. — howling cat, Jan 21 '19 at 12:18
See https://unix.stackexchange.com/questions/396149/globally-setting-cpu-affinity , you'll perhaps run into issues if you need to do this dynamically, as you'll have to re-write a config file before systemd starts. — nos, Jan 21 '19 at 15:17
@nos this is exactly my question: how do I re-write the systemd config file before it starts... — howling cat, Jan 21 '19 at 15:53
@howlingcat afaik. you can't, perhaps you rather need the application to figure it out and have it set its own affinity mask. — nos, Jan 21 '19 at 18:43

score 0 · Answer 1 · edited Jun 20 '20 at 09:12

0

How can we taskset all userland processes spawned by systemd during boot?

systemd.exec

Unit configuration files for services, sockets, mount points, and swap devices share a subset of configuration options which define the execution environment of spawned processes.

CPUAffinity=

Controls the CPU affinity of the executed processes. Takes a list of CPU indices or ranges separated by either whitespace or commas. CPU ranges are specified by the lower and upper CPU indices separated by a dash. This option may be specified more than once, in which case the specified CPU affinity masks are merged. If the empty string is assigned, the mask is reset, all assignments prior to this will have no effect

Alternatively, make a systemd unit that creates your topology configuration file and terminates. Make your other service units depend on that unit, so that they start after the topology configuration was created. Then make your services read the topology files and set cpusets and affinities accordingly on start (can be done by a wrapper script).

edited Jun 20 '20 at 09:12

Community

1
1

answered Jan 21 '19 at 14:53

Maxim Egorushkin

131,725
17
180
271

what I need is to be able to modify this parameter BEFORE systemd starts, because I do not know what architecture I may encounter. – howling cat Jan 21 '19 at 16:17
@howlingcat You can do that by modifying the configuration file before `systemd` starts. – Maxim Egorushkin Jan 21 '19 at 16:19
who starts it? what script loads it? – howling cat Jan 22 '19 at 06:39
@howlingcat You may find the 2nd method more practical. – Maxim Egorushkin Jan 22 '19 at 10:15
Almost, but not quite. What was missing for me was how to cause systemd shift itself and the userland processes it spawns away from the cores I wish to isolate. Anyway, see my answer above. – howling cat Jan 23 '19 at 07:39

howling cat · Answer 2 · 2019-01-22T19:56:05.370

I finally got things working correctly. Here is how:

Create a script at a path of your location. I named mine set-system-affinity.sh. Its contents should include:

#!/bin/bash
sed -i '/CPUAffinity/c\CPUAffinity=<whatever cpus should be configured>' /etc/systemd/system.conf
systemctl daemon-reexec

Create a service file at /etc/systemd/system. I named mine set-affinity.service.

Configure the service as follows:

[Unit]
Before=systemd-sysctl.service
DefaultDependencies=no

[Service]
Type=oneshot
ExecStart=/the/path/to/your/script/set-system-affinity.sh

[Install]
WantedBy=sysinit.target

Execute sudo systemctl enable set-affinity.service

If you are running a ramfs system like I do these scripts will need to be made part of the image that is loaded during boot, and the service be enabled somehow in the image.

How to start a service very early with SYSTEMD

How to set the Linux init process affinity without using grub?

2 Answers2