0

To submit jobs to a cluster through slurm, I can specify how many CPUs I want for a job with #SBATCH --ntasks-per-node={cpus}. However, this will send the job to any node with at least this many CPUs. This is normally fine, but say I'm on a cluster with nodes that have 24 CPUs and 40 CPUs. If I submit a 24 core job, some of them will go to the 40 CPU nodes, blocking further calculations on these nodes (unless they happen to ask for 16 or less).

Is there a way I can specify that I want nodes with at most a certain number of CPUs? I know it is possible to submit the job with a list of the nodes that I would accept, but I was hoping there was a way to do this without manually inserting this list into a variety of different job scripts.

Tyberius
  • 625
  • 2
  • 12
  • 20

1 Answers1

2

You can try using the --extra-node-info parameter to select nodes with a specific CPU configuration.

If the 24-CPUs nodes have 2 sockets with 12 cores and 1 hardware thread per core (i.e. hyperthreading or equivalent is disabled) you can set

--extra-node-info=2:12:1

The three numbers can also be set in separation options:

--sockets-per-node=<sockets>
--cores-per-socket=<cores>
--threads-per-core=<threads>

So if the differentiating factor between the 24-core nodes and the 40-core nodes is the number of sockets (e.g. 2 vs 4), use --sockets-per-node=2. If the differentiating factor is rather the number of cores per socket, use --cores-per-socket=12.

Note that these options are used to restrict the type of node to use, but they do not specify the size of the allocation. So the --ntasks-per-node parameter is still needed in conjunction with the options above.

If you do now know the CPU structure of the nodes, use the sinfo command:

sinfo -Nl

and look for the S:C:T column.

damienfrancois
  • 52,978
  • 9
  • 96
  • 110
  • Thank you for the answer. I'm not sure this will work in my case. It seems like the `--cores-per-socket` is still a lower bound, so it can find a node with more. In my case, the 24 core nodes are `2:12:1` and the 40 core nodes are `2:20:1`, so the job would still fill into a 40 core node even if I requested `--cores-per-socket=12`. The same seems to true for a smaller node: trying to get 20 core node with `2:10:1` using `--cores-per-socket=10` would allow it to go to a 24 or 40 core node. – Tyberius Jan 02 '21 at 21:15