2

I want to observe the performance of my MPI program using time command in linux. It shows only real, user and sys values for the program. However, I should examine what happens on each process. So, is there a way to see how long does my program take for each process separately?

simon_tulia
  • 396
  • 1
  • 6
  • 22
  • Are you constrained to use the `time` command (or other OS commands)? If you are able to change the code it is a rather simple matter to use `MPI_Wtime`, which would probably be more accurate as well. – R_Kapp Nov 03 '15 at 13:42
  • I've used the solution below. Thank you very much. – simon_tulia Nov 04 '15 at 05:44

1 Answers1

4

If you prepend time to the mpiexec/mpirun command, it will show you statistics about the command itself:

$ /usr/bin/time -p mpiexec -n 3 sleep 1
real 3.54
user 0.42
sys 0.15

To have it report statistics about each MPI rank, you should prepend time to the name of the MPI executable:

$ mpiexec -n 3 /usr/bin/time -p sleep 1
real 1.00
user 0.00
sys 0.00
real 1.00
user 0.00
sys 0.00
real 1.00
user 0.00
sys 0.00

In order to separate the output from all ranks and determine which is which and from which rank a measurement comes you can use the following with Open MPI:

$ mpiexec --tag-output -n 3 /usr/bin/time -p sleep 1
[1,1]<stderr>:real 
[1,1]<stderr>:1.00
[1,1]<stderr>:user 0.00
[1,1]<stderr>:sys 0.00
[1,0]<stderr>:real 1.00
[1,0]<stderr>:user 0.00
[1,0]<stderr>:sys 0.00
[1,2]<stderr>:real 1.00
[1,2]<stderr>:user 0.00
[1,2]<stderr>:sys 0.00

Each line is now prefixed by [1,rank]<stream>:. You can also write a wrapper named mpitime.sh that calls time with the option to write the information to a text file, e.g.:

#!/bin/sh
/usr/bin/time -o timing.$OMPI_COMM_WORLD_RANK $*

Running with mpiexec -n 3 mpitime.sh sleep 1 will produce three files: timing.0, timing.1, and timing.2, containing the timing for ranks 0, 1, and 2. The wrapper script has to be adapted for other implementations. See here.

Hristo Iliev
  • 72,659
  • 12
  • 135
  • 186