Safety guarantee for asynchronous sends in MPI

Question

In my application I use MPI to distribute jobs in a master-slave fashion. A job is given to a slave and then results are collected. In a multi-threaded version of this program, there was a potential deadlock when all processors simultaneously try to Send (blocking), because there was no matching Recv. I came with up with a solution which seems to work, but I would like to have the guarantee (other than by testing it another ten thousand times).

The safety of my program is certain, if this little code is guaranteed to be -- provided a conforming implementation. (obviously this works for two processors only and is not intended for more):

#include <cassert>
#include "mpi.h"

int main()
{
   MPI::Init();
   int ns[] = {-1, -1};
   int rank = MPI::COMM_WORLD.Get_rank();
   ns[rank] = rank;
   MPI::Request request = MPI::COMM_WORLD.Isend(&rank, sizeof(int), MPI::BYTE, 1 - rank, 0);
   MPI::COMM_WORLD.Recv(&ns[1 - rank], sizeof(int), MPI::BYTE, 1 - rank, 0);
   request.Wait();
   assert( ns[0] == 0 );
   assert( ns[1] == 1 );
   MPI::Finalize();
}

So my question is: Is the interleaving of an Isend with a Recv until I call Wait on the Request returned by Isend a well-defined safe thing in MPI?

(Disclaimer: This piece of code is not designed to be exception-safe or particularly beautiful. It's for demo purposes only)

score 1 · Accepted Answer · answered Mar 04 '14 at 15:38

Your code is perfectly safe. This is guaranteed by the semantics of the non-blocking operations as defined in the MPI standard §3.7.4 - Semantics of Nonblocking Communications:

Progress A call to MPI_WAIT that completes a receive will eventually terminate and return if a matching send has been started, unless the send is satised by another receive. In particular, if the matching send is nonblocking, then the receive should complete even if no call is executed by the sender to complete the send. Similarly, a call to MPI_WAIT that completes a send will eventually return if a matching receive has been started, unless the receive is satised by another send, and even if no call is executed to complete the receive.

A blocking operation in that context is equivalent to initiating a non-blocking one, immediately followed by a wait.

If the words of the standard are not reassuring enough, then this code section from the implementation of MPI_SENDRECV in Open MPI might help:

if (source != MPI_PROC_NULL) { /* post recv */
    rc = MCA_PML_CALL(irecv(recvbuf, recvcount, recvtype,
                            source, recvtag, comm, &req));
    OMPI_ERRHANDLER_CHECK(rc, comm, rc, FUNC_NAME);
}

if (dest != MPI_PROC_NULL) { /* send */
    rc = MCA_PML_CALL(send(sendbuf, sendcount, sendtype, dest,
                           sendtag, MCA_PML_BASE_SEND_STANDARD, comm));
    OMPI_ERRHANDLER_CHECK(rc, comm, rc, FUNC_NAME);
}

if (source != MPI_PROC_NULL) { /* wait for recv */
    rc = ompi_request_wait(&req, status);
} else {
    if (MPI_STATUS_IGNORE != status) {
        *status = ompi_request_empty.req_status;
    }
    rc = MPI_SUCCESS;
}

It doesn't matter if you use Irecv / Send / Wait(receive) or Isend / Recv / Wait(send) - both are equally safe when it comes to possible deadlocks. Of course deadlocks could (and will) occur if the interleaved operation is not properly matched.

The only thing that brings your code to non-conformance is the fact that it uses the C++ MPI bindings. Those were deprecated in MPI-2.2 and deleted in MPI-3.0. You should use the C API instead.

How do you guarantee that Wait is called in the example code? Both processes call a blocking Recv before Wait. — kraffenetti, Mar 04 '14 at 18:58
It's either that a background progression thread takes care for the progression of the non-blocking operation or (in single-threaded implementations) the blocking receive also progresses the non-blocking send. — Hristo Iliev, Mar 05 '14 at 09:20

Wesley Bland · Answer 2 · 2014-02-27T22:18:56.193

-1

This code is not guaranteed to work. MPI implementations are free to not do anything related to your Isend until you call the corresponding Wait function.

The better option is to make both your Send and Recv functions nonblocking. You would use Isend and Irecv and instead of using Wait, you would use Waitall. The Waitall function takes an array of MPI requests and waits on all of them (while making progress on each) simultaneously.

So your program would look like this:

#include <cassert>
#include "mpi.h"

int main()
{
   MPI::Init();
   int ns[] = {-1, -1};
   int rank = MPI::COMM_WORLD.Get_rank();
   MPI::Request requests[2];
   ns[rank] = rank;
   requests[0] = MPI::COMM_WORLD.Isend(&rank, sizeof(int), MPI::BYTE, 1 - rank, 0);
   requests[1] = MPI::COMM_WORLD.Irecv(&ns[1 - rank], sizeof(int), MPI::BYTE, 1 - rank, 0);
   MPI::Request::Waitall(2, requests)
   assert( ns[0] == 0 );
   assert( ns[1] == 1 );
   MPI::Finalize();
}

edited Feb 27 '14 at 22:18

answered Feb 27 '14 at 21:51

Wesley Bland

8,816
3
44
59

Interesting. But this seems weird to me: This looks like it is OK to put several `Isend` / `IRecv`s and then just wait for all of them at the end. Is this actually safe? – stefan Feb 27 '14 at 22:08
The only thing that can make it unsafe is the data. You can't modify the buffer that you're passing into the `Send` function until you've finished the `Wait`. Similarly, you can't read from the buffer you pass to `Recv` until you finish the `Wait`. – Wesley Bland Feb 27 '14 at 22:11
Oh my, seems like I've been using MPI way too complicated. Thank you! – stefan Feb 27 '14 at 22:13
The statement in your first paragraph is not true. The MPI standard makes it **explicitly clear** (§3.7.4) that a wait on a non-blocking operation (or a blocking operation respectively) should complete even when matched with a non-blocking operation that is still not waited on, therefore `MPI_ISEND / MPI_RECV / MPI_WAIT(send)` should never deadlock. And `MPI_Irecv / MPI_Send / MPI_Wait` is exactly how Open MPI implements the `MPI_Sendrecv` call. – Hristo Iliev Mar 04 '14 at 15:30

Safety guarantee for asynchronous sends in MPI

2 Answers2

Linked