3

I have been coding pretty simple MPI programs in C. I am not very good at C, but pretty confident in Python. I came to know that Python has MPI bindings. I am using Sun Grid Engine as DRMS (Distributed Resource Management System). Before jumping into python MPI, i would like to know if it is possible to submit python mpi program to Sun Grid Engine to harness the power of all available distributed processors of the system.

So, is it possible to submit python mpi program to Sun Grid Engine ?

World
  • 2,007
  • 7
  • 27
  • 37

1 Answers1

2

Yes it is possible and it isn't any different to running compiled MPI code with mpirun. All the usually cavaets about making sure the executable, libraries and support files your code needs to run on each node apply equally to python as to a compiled MPI application.

Grid Engine itself (or at least the pre-Oracle, pre-commercial versions I have used) doesn't know anything about MPI, per se, so after you job has gotten slot allocations from the resource manager your SGE job script probably needs to create its own machine file (or run a startup script) for the run and do whatever other incantations are necessary to ready the nodes which will run your code for the execution. The old SGE wiki and mailing lists had lots of useful examples of setting up an MPI machine file inside a running SGE job script. The inner details of this are almost completely MPI flavour specific and mostly independent of SGE, so it is a bit hard to provide specifics, but finally the command to run your Python application within the job file can be as simple as

mpirun <options here> python yourapp.py

or

mpirun <options here> yourapp.py

if yourapp.py has executable permissions and a suitable shebang line in it. I have run some rather large mpi4py applications with MPICH2 and SGE without difficulty doing the latter.

talonmies
  • 70,661
  • 34
  • 192
  • 269