1

I'm having trouble getting my head around the way jobs are launched by SLURM from an sbatch script. It seems like SLURM is ignoring the --ntasks argument and launching all the srun tasks in my batch file immediately. Here is an example, using a slight modification of the code from this answer on StackOverflow:

$ salloc --ntasks=1 --ntasks-per-core=1
salloc: Granted job allocation 1172
$ srun -n 1 sleep 10 & time srun -n 1 echo ok
[1] 5023
srun: cluster configuration lacks support for cpu binding
srun: cluster configuration lacks support for cpu binding
ok

real    0m0.052s
user    0m0.004s
sys 0m0.012s

So on my setup the srun echo command is being run immediately, whereas I would expect it to run after the srun sleep 10 command finishes.

I am using SLURM 2.6.5 to schedule and submit jobs on my personal workstation with 8 cores, and I installed it myself—so it's entirely possible the configuration is borked. Here are some relevant parts from the slurm.conf file:

# SCHEDULING
SchedulerType=sched/backfill
SelectType=select/cons_res
SelectTypeParameters=CR_CPU
# COMPUTE NODES
NodeName=Tom NodeAddr=localhost CPUs=7 RealMemory=28100 State=UNKNOWN
PartitionName=Tom Nodes=Tom Default=YES MaxTime=INFINITE State=UP

Here is the output from printenv | grep SLURM after running salloc --ntasks=1

SLURM_NODELIST=Tom
SLURM_NODE_ALIASES=(null)
SLURM_MEM_PER_CPU=4100
SLURM_NNODES=1
SLURM_JOBID=1185
SLURM_NTASKS=1
SLURM_TASKS_PER_NODE=1
SLURM_JOB_ID=1185
SLURM_SUBMIT_DIR=/home/tom/
SLURM_NPROCS=1
SLURM_JOB_NODELIST=Tom
SLURM_JOB_CPUS_PER_NODE=1
SLURM_SUBMIT_HOST=Tom
SLURM_JOB_NUM_NODES=1

I'd appreciate any comments or suggestions. Please let me know if any more info is required.

Thanks for reading,

Tom

Update after playing around some more

I have made some progress but I'm still not quite getting the behaviour I want.

If I use --exclusive I can get the echo step to wait for the sleep step:

salloc --ntasks=1
salloc: Granted job allocation 2387
srun -n 1 --exclusive sleep 10 & time srun -n 1 --exclusive echo ok
[1] 16602
ok
[1]+  Done                    srun -n 1 --exclusive sleep 10

real    0m10.094s
user    0m0.017s
sys 0m0.037s

and

salloc --ntasks=2
salloc: Granted job allocation 2388
srun -n 1 --exclusive sleep 10 & time srun -n 1 --exclusive echo ok
[1] 16683
ok

real    0m0.067s
user    0m0.005s
sys 0m0.020s

But I still don't know how to do this properly if I'm running a multi-step job where each step needs several processors, e.g.

salloc --ntasks=6
salloc: Granted job allocation 2389
srun -n 2 --exclusive stress -c 2 &
srun -n 2 --exclusive stress -c 2 &
srun -n 2 --exclusive stress -c 2 &

will give me 12 stress processes, as will

salloc --ntasks=6
salloc: Granted job allocation 2390
srun -n 1 --exclusive stress -c 2 &
srun -n 1 --exclusive stress -c 2 &
srun -n 1 --exclusive stress -c 2 &
srun -n 1 --exclusive stress -c 2 &
srun -n 1 --exclusive stress -c 2 &
srun -n 1 --exclusive stress -c 2 &

So what should I do if I want my sbatch script to take 6 processors and start three steps at a time, each with 2 processors? Is it correct to use srun --exclusive -n 1 -c 2 stress -c 2?

Community
  • 1
  • 1
Tom Harrop
  • 678
  • 2
  • 7
  • 23
  • 1
    Do you have hyperthreading enabled? Try removing the `--ntasks-per-core=1` part. Also, can you share the output of ``printenv|grep SLURM`` after you run ``salloc``? – damienfrancois Jan 13 '15 at 20:43
  • @damienfrancois, thanks for the reply. The processors are E5-2609 which don't have hyperthreading. Removing `--ntasks-per-core=1` didn't help, but I've edited my question to include output of `printenv | grep SLURM`. – Tom Harrop Jan 14 '15 at 08:26
  • Although removing `--ntasks-per-core=1` does eliminate the `cluster configuration lacks support for cpu binding` message, which is nice! – Tom Harrop Jan 14 '15 at 08:32
  • Can you share the full content of `scontrol show config` ? I can't see anything wrong otherwise. – damienfrancois Jan 14 '15 at 19:27
  • @damienfrancois Sure, [I put it on pastebin](http://pastebin.com./YL9ydHGk).Thanks again for helping. – Tom Harrop Jan 15 '15 at 08:55
  • Hum all seems ok. All I can suggest is to change ``CR_CPU`` to ``CR_CORE`` and use the output of ``slurmd -C`` in the definition of the node in ``slurm.conf``. The notion of ``CPU`` in Slurm is ambiguous and can correspond to sockets but I am unsure about the problem here. – damienfrancois Jan 15 '15 at 11:03
  • Yeah, it's been irritating me on and off for several months. I wonder if it's a bug or just a misconfiguration. It's a bit of a problem, because when I have a script that is going to run a whole bunch of tasks, I can't tell it how many tasks it's allowed at once. This is why I have `CPUs=7` even though there are actually 8 CPUs—otherwise scripts that run >7 tasks take over the whole computer... – Tom Harrop Jan 15 '15 at 12:44
  • I am having the same problem with SLURM 14.11.7 installed from source files and configured following the instructions posted on many websites. It also seems to be completely ignoring the --ntasks arguments and jobs can use more CPUs than requested. Did you manage to solve your problem? Any help would be much appreciated! – remek Jun 19 '15 at 13:45
  • @remek I'm afraid not. Do you think it's a bug? I didn't report it because I'm still on an old(er) version. If you end up reporting it, share the link here! – Tom Harrop Jun 19 '15 at 17:47
  • @TomHarrop Not sure it's a bug. Since I am new to slurm I would rather assume I did not configure something correctly. My problem is actually slightly different, although related, so I started a new post (http://superuser.com/questions/930054/slurm-allows-jobs-using-more-cpus-than-requested-to-start). Contrary to you, my jobs are actually queued and started only when CPUs are available. But there seems to be a discrepancy between the number of CPUs requested in the batch file and what is actually used by my jobs. – remek Jun 20 '15 at 10:50
  • I don't know, I'm not an expert but to me it looks like the same problem. Please post again if you get an answer over on SU. – Tom Harrop Jun 22 '15 at 06:20

1 Answers1

2

I think the missing pieces were the --exclusive and --cpus-per-task arguments. I get the behaviour I'm looking for with

salloc --ntasks=6
salloc: Granted job allocation 2457
srun --exclusive --ntasks=1 --cpus-per-task=2 stress -c 2 &
srun --exclusive --ntasks=1 --cpus-per-task=2 stress -c 2 &
srun --exclusive --ntasks=1 --cpus-per-task=2 stress -c 2 &
srun --exclusive --ntasks=1 --cpus-per-task=2 stress -c 2 &

This will launch 6 stress processes; the 4th stress command waits its turn at the end.

This may be obvious but it took a while to figure out!

Tom Harrop
  • 678
  • 2
  • 7
  • 23