2

I'm managing a PBS/torque HPC cluster, and now I'm setting up another cluster with SLURM. On the PBS cluster, I can set a queue to accept only interactive jobs by qmgr -c "set queue interactive_q disallowed_types = batch" and to accept only batch jobs by qmgr -c "set queue cu disallowed_types = interactive".

However, I cannot find the equivalent setups with SLURM after browsing through the official SLURM documents.

How can I set up a partition on SLURM cluster to only accept interactive or batch jobs?

Peter Mortensen
  • 2,318
  • 5
  • 23
  • 24
wdg
  • 153
  • 1
  • 5

1 Answers1

3

As of Slurm 20, there isn't any direct way to separate interactive jobs from batch jobs in a partition.

However, it is possible to use a job submit plugin which can distinguish the two in several ways; batch jobs have a job script (job_desc.script in Lua) associated with them where interactive jobs don't. Some interactive jobs have a PTY. I think there also may be a flag set that is new in Slurm 20.

Once the job submit plugin has identified a job as interactive or batch, it can either reject it for the given partition, or it can possibly change the partition it is being sent to, or it could modify the job in some other appropriate way (like assigning an "interactive" license to it or change the QOS) before final submission.

If you modify the job, then additional Slurm configuration can affect the job based on that.

user10489
  • 584
  • 1
  • 3
  • 14
  • This is indeed a good idea. Job submitted by `sbatch` will have its field `BatchFlag` set to 1. This is great, but according to https://github.com/SchedMD/slurm/blob/master/src/plugins/job_submit/lua/job_submit_lua.c, it does not seem that the lua plugin can access `BatchFlag` field. Writing C plugin is too troublesome, as the compilation of SLURM is done by the vendor, not me. Right now, I can only tell the job is interactive if `job_desc.argv[job_desc.argc-1]=="bash"`. This is certainly not sufficient. – wdg Jan 21 '22 at 09:17
  • 1
    I wrote my lua job submit plugin for slurm 19 or earlier, and checking for the script works fine, so the batchflag field is redundant anyway. Maybe they'll add it to a future version of the lua plugin. Look at `job_desc.script` – user10489 Jan 21 '22 at 12:25
  • Did you mean that for interactive jobs, `job_desc.script` is emtpy, but for batch jobs, they are not empty? – wdg Jan 21 '22 at 14:28
  • 1
    Right. Interactive jobs have no script and `job_desc.script` will be empty / not set. – user10489 Jan 22 '22 at 00:03