MPI RMA operation: Ordering between MPI_Win_free and local load

Question

I'm trying to do a simple test on MPI's RMA operation using MPI_Win_lock and MPI_Win_unlock. The program just let process 0 to update the integer value in process 1 and display it.

The below program runs correctly (at least the result seems correct to me):

#include "mpi.h"
#include "stdio.h"
#define root 0

int main(int argc, char *argv[])
{
  int myrank, nprocs;
  int send, recv, err;
  MPI_Win nwin;
  int *st;

  MPI_Init(&argc, &argv);
  MPI_Comm_size(MPI_COMM_WORLD, &nprocs);
  MPI_Comm_rank(MPI_COMM_WORLD, &myrank);

  MPI_Alloc_mem(1*sizeof(int), MPI_INFO_NULL, &st);

  st[0] = 0;
  if (myrank != root) {
    MPI_Win_create(st, 1*sizeof(int), sizeof(int), MPI_INFO_NULL, MPI_COMM_WORLD, &nwin);
  }
  else {
    MPI_Win_create(NULL, 0, sizeof(int), MPI_INFO_NULL, MPI_COMM_WORLD, &nwin);
  }

  if (myrank == root) {
    st[0] = 1;
    MPI_Win_lock(MPI_LOCK_EXCLUSIVE, 1, 0, nwin);
    MPI_Put(st, 1, MPI_INT, 1, 0, 1, MPI_INT, nwin);
    MPI_Win_unlock(1, nwin);
    MPI_Win_free(&nwin);
  }
  else { // rank 1
    MPI_Win_free(&nwin);
    printf("Rank %d, st = %d\n", myrank, st[0]);
  }

  MPI_Free_mem(st);
  MPI_Finalize();
  return 0;
}

The output I got is Rank 1, st = 1. But curiously, if I switch the lines in the else block for rank 1 to

  else { // rank 1
    printf("Rank %d, st = %d\n", myrank, st[0]);
    MPI_Win_free(&nwin);
  }

The output is Rank 1, st = 0.

I cannot find out the reason behind it, and why I need to put MPI_Win_free after loading the data is originally I need to put all the stuff in a while loop and let rank 0 to determine when to stop the loop. When condition is satisfied, I try to let rank 0 to update the flag (st) in rank 1. I try to put the MPI_Win_free outside the while loop so that the window will only be freed after the loop. Now it seems that I cannot do this and need to create and free the window every time in the loop?

Welcome! Usually we try to avoid thank taglines in the questions to keep them tidier, you can read why in here: "[Should 'Hi', 'thanks,' taglines, and salutations be removed from posts?](http://meta.stackexchange.com/questions/2950/should-hi-thanks-taglines-and-salutations-be-removed-from-posts)". — veducm, Jan 28 '14 at 10:40

score 0 · Accepted Answer · edited May 23 '17 at 11:49

I'll be honest, MPI RMA is not my speciality, but I'll give this a shot:

The problem is that you're running into a race condition. When you do the MPI_PUT operation, it sends the data from rank 0 to rank 1 to be put into the buffer at some point in the future. You don't have any control over that from rank 0's perspective.

One rank 1's side, you're not doing anything to complete the operation. I know that RMA (or one-sided operations) sound like they shouldn't require any intervention on the target side, but the do require a bit. When you use one-sided operations, you have to have something on the receiving side that also synchronizes the data. In this case, you're trying to use MPI put/get operations in combination with non-MPI load store operations. This is erroneous and results in the race condition you're seeing. When you switch the MPI_WIN_FREE to be first, you complete all of the outstanding operations so your data is correct.

You can find out lots more about passive target synchronization (which is what you're doing here) with this question: MPI with C: Passive RMA synchronization.

According to the MPI standard (§11.4.3, p.357), the Put operation should be completed when MPI_Win_lock returns, so the cause may not be on the window. The Example 11.12 Rule 6 (p.367) actually shows a similar situation. The "tricky" point is the use of MPI_Barrier, which tries to guarantee the execution order. I modified my code below and somehow it works: — Leo, Jan 29 '14 at 10:11
Right, but the problem is that rank 1 isn't around long enough to received the data that has been out when you're printing the value. When you have the free first, you force it to synchronize because it is a collective call over all processes in the window. — Wesley Bland, Jan 29 '14 at 12:40

MPI RMA operation: Ordering between MPI_Win_free and local load

1 Answers1