0

How do you go about running the same program multiple times but with different arguments each instance on a cluster, submitted through a PBS. Also, is it possible to designate each of these programs to a separate node? Currently, if I have a PBS with the following script:

#PBS -l nodes=1:ppn=1
/myscript

it will run the single program once, on a single node. If I use the following script:

#PBS -l nodes=1:ppn=1
/mscript -arg arg1 &
/myscript -arg arg2

I believe this will run each program in serial, but it will use only one node. Can I declare multiple nodes and then delegate specific ones out to each instance of the program I wish to run?

Any help or suggestions will be much appreciate. I apologize if I am not clear on anything or am using incorrect terminology...I am very new to cluster computing.

dsolimano
  • 8,870
  • 3
  • 48
  • 63
abovezero
  • 63
  • 2
  • 11

2 Answers2

0

You want to do that using a form of MPI. MPI stands for message passing interface and there are a number of libraries out there that implement the interface. I would recommend using OpenMPI as it integrates very well with PBS. As you say you are new, you might appreciate this tutorial.

dbeer
  • 6,963
  • 3
  • 31
  • 47
0

GNU Parallel would be ideal for this purpose. An example PBS script for your case:

#PBS -l nodes=2:ppn=4 # set ppn for however many cores per node on your cluster
#Other PBS directives

module load gnu-parallel   # this will depend on your cluster setup

parallel -j4 --sshloginfile $PBS_NODEFILE /mscript -arg {} \
  ::: arg1 arg2 arg3 arg4 arg5 arg6 arg7 arg8

GNU Parallel will handle ssh connections to the various nodes. I've written out the example with arguments on the command line, but you'd probably want to read the arguments from a text file. Here are links to the man page and tutorial. Option -j4 should match the ppn (number of cores per node).

Steve Koch
  • 912
  • 8
  • 21