1

I'm developing a MPI+cuda project and I've tried to profile my app with nvvp and nvprof but, in both cases it doesn't give a profile. The app works completely fine, but no profile is generated. 

nvprof mpirun -np 2 MPI_test
[...]
======== Warning: No CUDA application was profiled, exiting

I tried with simpleMPI cuda example with the same result.

I'm using CUDA 5.0 in a 580 GTX and openMPI 1.7.3 (featured, not release yet because I'm testing the CUDA-aware option)

Any ideas? Thank you very much.

jgonzac
  • 175
  • 2
  • 9
  • 2
    Are you try to profile under linux? Because on linux os I have to pass to complete absolute path to the application to nvprof. If I don't pass it, than I get the same warning like you. – hubs Dec 18 '13 at 14:55

1 Answers1

2

mpirun itself is not a CUDA application. You have to run the profiler like mpirun -np 2 nvprof MPI_test. But you also have to make sure that each instance of nvprof (two instances in that case) is writing to a different output file. Open MPI exports the OMPI_COMM_WORLD_RANK environment variable that gives the process rank in MPI_COMM_WORLD. This could be used in just another wrapper, e.g. wrap_nvprof:

#!/bin/bash
nvprof -o profile.$OMPI_COMM_WORLD_RANK $*

This should be run like mpirun -n 2 ./wrap_nvprof executable <arguments> and after it has finished there should be two output files with profile information: profile.0 for rank 0 and profile.1 for rank 1.

Edit: There is an example nvprof wrapper script that does the same in a more graceful way and that handles both Open MPI and MVAPICH2 in the nvvp documentation. A version of the script is reproduced in this answer to a question that yours is more or less a duplicate of.

Community
  • 1
  • 1
Hristo Iliev
  • 72,659
  • 12
  • 135
  • 186
  • I had already tested the script in the answer you say. But the problem was the path to the executable, nvprof needs ./MPI_test so `mpirun -n 2 ./wrap_nvprof /path_to_executable ` is the correct way. Thank you very much! – jgonzac Dec 19 '13 at 09:02
  • It depends on how `nvprof` executes the binary. Relative paths, e.g. `./executable`, should probably work too. – Hristo Iliev Dec 19 '13 at 11:37
  • Yes, of course, I mean, `nvprof` needs a `./` or `/` before executable, at least in my case. – jgonzac Dec 19 '13 at 13:52