4

i wanted to submit a multi-threaded job to the cluster network i'm working with - but the man page about qsub is not clear how this is done - By default i guess it just sends it as a normal job regardless of the multi-threading - but this might cause problems, i.e. sending many multi-threaded jobs to the same computer, slowing things down.

Does anyone know how to accomplish this? thanks.

The batch server system is sge.

dan12345
  • 1,594
  • 4
  • 20
  • 30

3 Answers3

5

In SGE/UGE the configuration is set by the administrator so you have to check what they've called the parallel environments

qconf -spl 
make
our_paraq

look for one with $pe_slots in the config

qconf -sp make
qconf -sp our_paraq

qsub with that environment and number of cores you want to use

qsub -pe our_paraq 8 -cwd ./myscript

If you're using mpi you have more choices for the config allocation rule ($pe_slots above) like $round_robin and $fill_up, but this should get you going.

Adi Lester
  • 24,731
  • 12
  • 95
  • 110
j_m
  • 81
  • 1
  • 3
0

The answer by the user "j_m" is very helpful, but in my case I needed to both request multiple cores AND submit my job to a specific node. After a copious amount of searching, I finally found a solution that worked for me and I'm posting it here so that other people who might have a similar problem don't have to go through the same pain (please note that I'm leaving this as an answer instead of a reply because I don't have enough reputation for making replies):

qsub -S /bin/sh -cwd -l h=$NODE_NAME -V -pe $ENV_NAME $N_OF_CORES $SCRIPT_NAME

I think the variables $NODE_NAME, $N_OF_CORES and $SCRIPT_NAME are pretty straightforward. You can easily find $ENV_NAME by following the answer by "j_m".

Seung-been Steven Lee
  • 31
  • 5
0

If your job is multithreaded, you can harness the advantage of multithreading even in SGE. In SGE single job can use one or many CPUs. If you submit a job that uses single processor and you have a program that have many threads than a single processor can handle, problem occurs. Verify how many processors your job is using and how many threads per CPU your program is creating.

In my case i have a java program that uses one processor with two threads, it works pretty efficiently. i submit same java program for execution to many CPUs with 2 threads each to make it parallel as i have not use MPI.

World
  • 2,007
  • 7
  • 27
  • 37