Race condition with MPI

Question

I am trying to implement Tournament barrier using MPI. Here, is the code I have written. I am writing only the arrival phase and wake up phase

//Arrival phase
while(1)
{
    if((!strcmp(round[my_id][round_num].role,"winner"))||(!strcmp(round[my_id][round_num].role,"champion")))
    {
       printf("%d is the winner of round %d\n",my_id,round_num);
       MPI_Recv(&reach_msg, sizeof(reach_msg), MPI_BYTE, round[my_id][round_num].opponent, tag, MPI_COMM_WORLD, &status);
       printf("%d received: %s\n",my_id,reach_msg);
    }

    else if(!strcmp(round[my_id][round_num].role,"loser"))
    {
       printf("%d is the loser of round %d\n",my_id,round_num);
       sprintf(reach_msg,"%d arrived at the barrier",my_id);
       MPI_Send(reach_msg,strlen(reach_msg+1),MPI_BYTE,round[my_id][round_num].opponent,tag,MPI_COMM_WORLD);
       MPI_Recv(wakeup_msg,sizeof(wakeup_msg),MPI_BYTE,round[my_id][round_num].opponent,tag,MPI_COMM_WORLD,&status);
       printf("%d received: %s\n",my_id,wakeup_msg);
    }

if(round_num==num_rounds)
       break;
    else
       round_num++;
}

printf("%d is out of arrival tree\n",my_id);

//wakeup tree
      while(1)
     {
       printf("%d prints: round num is: %d\n",my_id,round_num);
       if(round_num==0)
            break;
       sprintf(wakeup_msg,"wakeup msg from %d of %d",my_id,P);

       if((!strcmp(round[my_id][round_num].role,"winner"))||(!strcmp(round[my_id][round_num].role,"champion")))
       MPI_Send(wakeup_msg,strlen(wakeup_msg+1),MPI_BYTE,round[my_id][round_num].opponent,tag,MPI_COMM_WORLD);
       round_num--;
      }

    MPI_Finalize();
    return 0;
   }

I dont understand why race conditions occur. I believe the MPI_Send and MPI_Recv are blocking functions. But, sometimes it doesn't behave that way

EDIT: Here is a sample output where race condition happens. As you can see, 0 received message("1 arrived at the barrier") from 1 even before 1 sent the message.

0 is the winner of round 1

0 received: 1 arrived at the barrie

0 is the winner of round 2

1 is the loser of round 1

1 sending reach msg

2 is the winner of round 1

2 received: 3 arrived at the barrie

2 is the loser of round 2

2 sending reach msg

3 is the loser of round 1

3 sending reach msg

0 received: 2 arrived at the barrie

0 sending wakeup msg

1 received: wakeup msg from 0 with tag at round 1

2 received: wakeup msg from 0 with tag at round 2

2 sending wakeup msg

3 received: wakeup msg from 2 with tag at round 1

By race condition, what i mean is that receiving process receives message even before sending process sends it ! — CuriousCoder, Oct 21 '11 at 06:02
It's fairly unlikely that your program is violation causality. You're just seeing it *printed* that way. IIRC MPI (though it probably depends on the implementation) does all it printing on process 0, which means that "printf" on process 1 actually sends it data for printing by process 0. — user786653, Oct 21 '11 at 06:41
@user786653: I am actually running this code in the same machine. I am allocated a cluster of machines. What if i distribute the code in different machines of the cluster and run the program? will the printf statements still be sent over to the process 0 for printing? — CuriousCoder, Oct 21 '11 at 06:46

score 2 · Answer 1 · answered Oct 21 '11 at 06:50

To my experience you most likely have a problem of observation and not of your underlying algorithm. Usually the printfs in such a setting simply arrive out-of-order. You'd have to

put time stamps in your output
write to different files, one per each MPI process
merge them back together by sorting according to your time stamp

score 1 · Accepted Answer · answered Oct 22 '11 at 00:21

For debugging mpi programs, print statements are generally not a very good solution. As the print statments need to be send to the console from different nodes over the network and are cached before.

To debug it you should let each processor write to an own output file. Or let only one processor print statements, where you disable caching by either using std::cerr or somehow disable the caching of printf.

score 0 · Answer 3 · answered Oct 21 '11 at 06:15

0

I am not sure I fully understand your problem... adding the output of the code to your question might help.

What I can say for sure is that MPI_Send and MPI_Recv are definitely blocking functions. Have you tried to use non-blocking functions instead (i.e. MPI_Isend and MPI_Irecv) ? If yes, did it solve your problem ?

answered Oct 21 '11 at 06:15

Bertrand Moreau

396
2
5

I have added the output. I didn't use non blocking send and receive bcoz i need blocking functionality – CuriousCoder Oct 21 '11 at 06:34
Thank you for adding the output. I agree with Jens and user786653, your problem is more likely to be related with the print order rather than with your implementation. – Bertrand Moreau Oct 21 '11 at 06:58

Race condition with MPI

3 Answers3