How to use ltrace for mpi programs?

Question

I want to know how to use ltrace to get library function calls of mpi application but simply ltrace doesn't work and my mpirun cannot succeed. Any idea?

Answer 1

You should be able to simply use:

$ mpiexec -n 4 -other_mpiexec_options ltrace ./executable

But that will create a huge mess since the outputs from the different ranks will merge. A much better option is to redirect the output of ltrace to a separate file for each rank. Getting the rank is easy with some MPI implementations. For example, Open MPI exports the world rank in the environment variable OMPI_COMM_WORLD_RANK. The following wrapper script would help:

#!/bin/sh

ltrace --output trace.$OMPI_COMM_WORLD_RANK $*

Usage:

$ mpiexec -n 4 ... ltrace_wrapper ./executable

This will produce 4 trace files, one for each rank: trace.0, trace.1, trace.2, and trace.3.

For MPICH and other MPI implementations based on it and using the Hydra PM exports PMI_RANK and the above given script has to be modified and OMPI_COMM_WORLD_RANK replaced with PMI_RANK. One could also write an universal wrapper that works with both families of MPI implementations.