My code terminates itself after running for a good number of cycles (in do loop) when parallel processing, how do I debug?

Question

The following error is obtained when I run the code via parallel processing:

Abort(1) on node 25 (rank 25 in comm 0): application called MPI_Abort(MPI_COMM_WORLD, 1) - process 25

I use the intel compiler in F90, with the following intel flags while compiling to debug: -g -traceback -check bounds.

If I run the code directly in one processor, then the flags print the issue that is causing my code to terminate. But when I use mpirun or mpiexec to run the same code in parallel, the flags don't print anything. My code terminates itself only at times and after long run times.

I would like to know what can be done to see print the flags output while using mpirun or mpiexec.

I want to print the flag outputs of -g -traceback -check outputs of an error while running the code in mpi.

Welcome, I suggest taking the tour. Which MPI library are you using? — Vladimir F Героям слава, Dec 11 '22 at 07:37
If the code is calling MPI_Abort -check bounds will be producing no outpt as it is the code detecting an error, not the runtime. For a traceback https://stackoverflow.com/questions/65157007/tracebackqq-with-ifort-leads-to-segmentation-fault/65157202#65157202 might help — Ian Bush, Dec 11 '22 at 08:26

score 1 · Answer 1 · answered Dec 19 '22 at 10:27

1

To debug MPI code, we need to use the below example command:

For Compilation: mpiifort -g -traceback -check bounds test.f90 -o test 
For Execution: mpirun -n 4 -gdb ./test

The above execution command will lead you to GDB debugging mode where you can start debugging your application.

For more information, refer to the below link: https://www.intel.com/content/www/us/en/develop/documentation/mpi-developer-guide-linux/top/debugging-applications.html

Thanks, Santosh

answered Dec 19 '22 at 10:27

Santosh-Intel

156
2
6

As Ian pointed out, the MPI_Abort is called by the program. In that case the run with gdb will exit normally and will not stop unless a breakpoint is set, e.g. at MPI_Abort. – Vladimir F Героям слава Dec 19 '22 at 10:49

My code terminates itself after running for a good number of cycles (in do loop) when parallel processing, how do I debug?

1 Answers1