1

I have SLURM setup on a single node, which is also a 'login node'. I would like to restrict interactive CPU usage, e.g. outside the scheduling system.

I found the following article which suggests to use cgroups for this: https://rolk.github.io/2015/04/20/slurm-cluster

I thought this was perfect and set it up in the following way:

/etc/cgconfig.conf:

group root {
        cpu {}
}

group interactive {
        cpu {
                cpu.cfs_period_us=100000;
                cpu.cfs_quota_us=400000;
        }
}

group batch {
        cpu {}
}

/etc/cgrules.conf:

root             cpu    root
slurm            cpu    batch
*                cpu    interactive

I use cgrulesengd to apply these rules.

The problem is that tasks properly queued using SLURM also run under the respective user's own user (and not slurm), so they get put in the interactive cgroup. Of course, this is not what I want. I assume the article I followed expected that the tasks would get run under the slurm user.

Is there a way to put all tasks run by SLURM in the right cgroup?

Compizfox
  • 384
  • 1
  • 6
  • 18

1 Answers1

1

I have been beating my head against exactly this problem for a while now and I finally came across this solution on stackexchange: https://unix.stackexchange.com/questions/526994/limit-resources-cpu-mem-only-in-ssh-session. My answer below is adapted from that. First, create your limited "interactive" cgroup in /etc/cgconfig.conf:

group interactive {
   perm {
      admin {
         uid=root;
         gid=root;
      }
      task {
         uid=root;
         gid=shelluser;
         fperm=775;
      }
   }
   cpu {
      cpu.cfs_period_us = "1000000";
      cpu.cfs_quota_us = "500000";
   }
   memory {
      memory.limit_in_bytes = "8G";
      memory.memsw.limit_in_bytes = "8G";
   }
}

We've added the perm{} subsection, which will enable user accounts to place themselves into this cgroup using cgclassify (I think, but do not know for sure, that they won't have permission to then remove themselves from that group, however that may be a loophole in this process. For my purposes, it's a potential loophole I'm willing to accept given my user base). The task{} section dictates who is allowed to assign tasks to the cgroup. In this case, we set the tasks file to group-writable and owned by the 'shelluser' group. You'll need your users to be members of this group or none of this will work.

Now, create the script /etc/profile.d/ssh.sh. In this script we'll assign ssh sessions to the 'interactive' cgroup, thus preventing users from hogging system resources directly. However, sometimes you do need to run some things interactively, so I've allowed interactive sessions spawned by slurm to run unrestricted. Again, the way I've done this is almost certainly exploitable by a clever user, but it's a risk I'm willing to accept given our users:

# check if user is in an SSH session and *hasn't* been spawned by slurm
if [[ -n $SSH_CONNECTION ]] && [[ -z $SLURM_JOBID ]]; then
  # get the shell pid
  SESSIONPID=$$
  # attach the shell to the relevant cgroup
  cgclassify -g memory,cpu:interactive $SESSIONPID
  # now everything spawned by this shell should be in this cgroup
fi

Users logging in via SSH will be placed in the 'interactive' cgroup on login (provided they're members of the 'shelluser' unix group). In that case they'll be limited to 8G and 1/2 of one CPU. If they need to be able run stuff directly without limits, they can spawn a bash instance under slurm with

$ srun -p <partition> -c <cpus> --mem=<mem> -t <time> --pty /bin/bash

And it should run unrestricted.

Luther Blissett
  • 111
  • 1