Open MPI ranks are not in order

Question

When i run an "Open MPI" program, it generally assigns ranks in random order I want to know is there a way to always assign ranks in order?

So instead of this

Hello, World. I am 2 of 3
Hello, World. I am 0 of 3
Hello, World. I am 1 of 3

can i get this

Hello, World. I am 0 of 3
Hello, World. I am 1 of 3
Hello, World. I am 2 of 3

EDIT

here is the code

    PROGRAM hello
INCLUDE 'mpif.h'
INTEGER*4 :: numprocs, rank, ierr
CALL MPI_INIT(ierr)
CALL MPI_COMM_SIZE(MPI_COMM_WORLD, numprocs, ierr)
CALL MPI_COMM_RANK(MPI_COMM_WORLD, rank, ierr)
write(*,*) 'Hello World. I am', rank, 'of', numprocs
CALL MPI_FINALIZE(ierr)
    END PROGRAM hello

I have tested it on i5 processor (4 threads) when i run

    mpirun -np 4 myprog

it works as i want it to, ranks printed in order 0-3, otherwise (like with 3 as shown above) it just wont do it (tested it like 100 time)

The mapping of ranks to hosts/cores is deterministic in every MPI implementation. The exact progress of each rank is (largely) independent during the course of the job. The "out of order" nature of the prints illustrates that each of the processes got to the write, and that write was transmitted through stdout back to the mpirun command for output to the terminal, in a different order. This is the fundamental nature of MPI programs. There is "loose" coordination between the ranks, at best. Synchronization across all the ranks is one of the fundamental problems in parallel programming. — Stan Graves, Dec 19 '13 at 01:55

score 3 · Accepted Answer · answered Dec 17 '13 at 12:01

The order in which ranks are assigned is never random with most MPI implementations and usually there are mechanisms to precisely control it. What is random is the order in which the output from the different ranks arrives at the MPI launcher (mpirun, mpiexec, etc.) through the IO redirection mechanism. Due to the buffering that's usually involved, one could never be sure that if e.g. rank 0 outputs some text before rank 1, then the output from rank 0 would necessarily arrive before the output from rank 1. The only portable way to guarantee ordering of the text output is to have all ranks channel their messages through a single rank that does IO.

With some implementations it might be possible to do something like linear token passing or a sequence of barriers, e.g.:

int rank, size;

MPI_Comm_rank(MPI_COMM_WORLD, &rank);
MPI_Comm_size(MPI_COMM_WORLD, &size);

for (int i = 0; i < size; i++)
{
   if (i == rank)
      printf("Hello world! I am %d of %d\n", rank, size);
   MPI_Barrier(MPI_COMM_WORLD);
}

The rationale behind such code is that MPI_Barrier could possibly progress pending communication operations before it completes, including those that carry the redirected standard output. Still there is no guarantee that the output from printf() is immediately shown in the console output of mpirun/mpiexec.

What if i have a loop from say 1 to 3560, i divide work in the loop by 5 processors, So each processor does 730 iterations of the loop. Now the ranks do not come in order (i have tried) and Rank 0 does iterations [731 - 1461] but i want it to do iterations [1 - 730] — Jovi DSilva, Dec 17 '13 at 15:38
Your question doesn't make sense. Just have your application do a check, `if (rank == 0) {do 1 - 730}`. — Wesley Bland, Dec 17 '13 at 16:16

Rob Latham · Answer 2 · 2017-09-05T13:52:35.770

3

Many mpi implementations will let you prefix the output with the rank of that process. then you can sort it after the fact. In MPICH, this is --prepend-rank or -l. In OpenMPI it is --tag-output

edited Sep 05 '17 at 13:52

answered Dec 18 '13 at 03:35

Rob Latham

5,085
3
27
44

Open MPI ranks are not in order

2 Answers2

Linked