Using printf with MPI leads to non-deterministic output

Question

The following code has non-deterministic behaviour on my machine (already with using only two processes).

#include <mpi.h>
#include <stdio.h>
#include <stdlib.h>

int main(int argc, char *argv[]) {
  int rank, size;
  MPI_Init(&argc, &argv);
  MPI_Comm_rank(MPI_COMM_WORLD, &rank);
  MPI_Comm_size(MPI_COMM_WORLD, &size);

  if (rank == size - 1) {
    printf("I am the last rank\n");
    MPI_Barrier(MPI_COMM_WORLD);
  } else {
    MPI_Barrier(MPI_COMM_WORLD);
    printf("I am rank %d\n", rank);
  }
  MPI_Finalize();
  return 0;
}

Sometimes the output from the last rank appears first on the terminal but sometimes it does appear later, even though a barrier is used.

I assume the reason for this is that printf does internal buffering and that MPI respectively mpirun/mpiexec and printf do not really cooperate with each other. Is there a more valid source to read up on this topic?

No, it's not stdio stream buffering. Adding `setlinebuf(stdout);` did not change the result. But, always putting the `printf` _after_ the `MPI_Barrier` call did seem to work. — Craig Estey, Nov 23 '22 at 14:55
If you got this to work it's only because you're running only on the cores of a single processor. As soon as you go to multiple nodes there is no way screen output can be synchronized. Don't even try. There are many questions about this already. — Victor Eijkhout, Nov 23 '22 at 15:33

score 1 · Accepted Answer · answered Nov 23 '22 at 18:34

Output redirection is strongly platform-dependent and is only tangentially mentioned in the MPI standard. There is absolutely no guarantee as to what order output from different ranks will be displayed in. There isn't even a guarantee that you can get that output - some implementations only redirect the output from rank 0, some redirect no output, and both cases are compliant with the MPI standard.

If you want strongly ordered output, you should only do output in a single rank.

Using printf with MPI leads to non-deterministic output

1 Answers1