sbatch slurm jobs which release CPU resources individually

Question

I have a cluster of many nodes with many cores and I simply want to run thousands of jobs that just require a single CPU each on them. Preferably with sbatch. After going through the documentation for several hours I still run into problems. My current setup is:

#SBATCH --nodes=4
#SBATCH --tasks-per-node=25
#SBATCH --distribution=block

srun ./my_experiment

I start several of these with sbatchand they seem to queue up nicely.

This script starts 100 instances of my_experiment which is intended. Unfortunately they seem to hog the resources of all 100 CPUs even if 99 experiments already ended. How do I alleviate this?

Secondly they don't seem to share nodes with each other. Even though the nodes have +40 cores.

Is it even possible to sbatch a bunch of tasks and have them release their resources individually?

How is Slurm configured w.r.t node sharing? (what is the output of `scontrol show config | grep select`?) — damienfrancois, Jan 22 '18 at 08:55
Editing error, I fixed it. The output is `SelectType = select/cons_res`. It is a big research cluster which I don't have the admin rights to — David Schumann, Jan 23 '18 at 10:15

score 1 · Accepted Answer · answered Jan 23 '18 at 12:47

Unfortunately they seem to hog the resources of all 100 CPUs even if 99 experiments already ended.

That is because you create one job, spanning at least 4 nodes, requesting 25 CPUs per node for 25 tasks. Jobs release their allocation at the end of all tasks.

Assuming there is no communication between your processes, your workflow seems better suited for job arrays. With job arrays, the idea is to create many jobs that are independent but easily manageable in sets.

#SBATCH --ntasks=1
#SBATCH --array=1-100

srun ./my_experiment

You will end up with 100 jobs, starting and ending independently one of another, but which you can kill with a single command.

If your program my_experiment used the SLURM_PROC_ID environment variable, you can replace it with the SLURM_ARRAY_TASK_ID .

Secondly they don't seem to share nodes with each other. Even though the nodes have +40 cores.

You require explicitly 25 cores per node for each job, so if the core count is not at least 50, Slurm cannot place two of such jobs on the same node. If the core count is larger than 50 it might be due to memory requirements.

Using exactly your script still only utilizes one node per task. Therefore only executing as many tasks as I have nodes at a time. (not as many tasks as I have CPUs). Is this a result of a misconfiguration by the admins? If yes, what setting could be the cause? — David Schumann, Jan 23 '18 at 14:05

sbatch slurm jobs which release CPU resources individually

1 Answers1