systemd deletes sub-cgroups started by other services

Question

I have a service (HTCondor batch system), which is started as service unit within cpu,cpuacct and memory cgroup slices (CentOS 7 @ 3.10.0-*).

The service starts sub-processes (~~> batch jobs) for which it creates sub-slices, i.e., subdividing its parent resources. Without further interfering, the started processes are in the sub-slices

wc -l /sys/fs/cgroup/cpu,cpuacct/system.slice/condor.service/tasks
  19
wc -l /sys/fs/cgroup/cpu,cpuacct/system.slice/condor.service/*/tasks
  29 /sys/fs/cgroup/cpu,cpuacct/system.slice/condor.service/condor_var_lib_condor_execute_slot1_2@batch0311.desy.de/tasks
  22 /sys/fs/cgroup/cpu,cpuacct/system.slice/condor.service/condor_var_lib_condor_execute_slot1_3@batch0311.desy.de/tasks
  22 /sys/fs/cgroup/cpu,cpuacct/system.slice/condor.service/condor_var_lib_condor_execute_slot1_4@batch0311.desy.de/tasks
  ...

and as cross-check, the processes have their corresponding cgroups also in their process info, e.g.,

cat /proc/58683/cgroup 
  11:perf_event:/
  10:memory:/system.slice/condor.service/condor_var_lib_condor_execute_slot1_6@batch0311.desy.de
  9:devices:/system.slice
  8:blkio:/system.slice/condor.service     /condor_var_lib_condor_execute_slot1_6@batch0311.desy.de
  7:cpuset:/
  6:freezer:/system.slice/condor.service/condor_var_lib_condor_execute_slot1_6@batch0311.desy.de
  5:hugetlb:/
  4:cpuacct,cpu:/system.slice/condor.service/condor_var_lib_condor_execute_slot1_6@batch0311.desy.de
  3:pids:/system.slice/condor.service
  2:net_prio,net_cls:/
  1:name=systemd:/system.slice/condor.service

AFAIS, systemd seems to be not aware of the sub-slices as systemd-cgls shows the processes directly beneath the the parent unit's cgroup

systemd-cgls
   ...
   ├─condor.service
   │ ├─  781 /bin/bash ...foo...
   │ ├─ 1596 condor_starter -f -a slot1_4 ...baz...

Now, when adding a new unit, reloading the systemd daemons and starting the new unit, all the job sub-cgroups disappear and their processes get attached to the parent cgroup.

wc -l /sys/fs/cgroup/cpu,cpuacct/system.slice/condor.service/tasks
  337 /sys/fs/cgroup/cpu,cpuacct/system.slice/condor.service/tasks

My assumption is, that systemd is not aware of the sub-slices (guessing from systemd-cgls), while from the kernel's view these are proper cgroup slices. When starting the new unit, systemd notices the discrepancy from its expectations and 'cleans up'.

Can this behaviour somehow be avoided?

score 1 · Answer 1 · answered Jul 05 '18 at 14:47

It looks like upstream since fixed this by specifying the Delegate= directive (commit 890186d82a – though specifying a subset of controllers would be a bit more elegant than simply true IMHO). If that update isn’t propagated to the CentOS package, you can apply it locally with the following command:

systemctl set-property condor.service Delegate=true

score 1 · Answer 2 · answered Jul 10 '18 at 07:41

problem was, that by default systemd assumes that all sub-cgroup/slices are handled by itself and that any unit processes have no own control.

When enabling delegation for a unit, systemd will not try to take control of the unit's sub-resources

[Service]
...
Delegate=true

(the [Slice] section might also be the right section, but apparently the right section depends on the release/kernel so #YMMV)

note that the cgroups/slices shown by systemd-cgls and systemd-cgtop still differ and only systemd-cgtop shows the 'right'Ä kernel view of cgroups while systemd-cgls does not show any sub-hierarchy of slices even with delegation)

systemd deletes sub-cgroups started by other services

2 Answers2