Safety guarantee for interleaved MPI Isend / Recv

Question

In a related question I learned that performing request = Isend(...); Recv(...); request.Wait(); is not guaranteed to work, as Isend may not do anything until request.Wait(), hence deadlocking at Recv(...) (see original question for details).

But what about if Isend() / Wait() is performed on another thread than Recv? I'm now not directly interested in safety guarantees by the standard. This is because the standard only asserts thread-safety, if the appropriate Init_thread method is called and returns the correct level. With my configuration of openMPI this isn't the case. However, I can't see a reason that an implementation actually restricts calls to only from the thread which called Init_thread (an actual comparison for the thread id would be necessary). My reasoning is: if I serialize all sends and all recvs, mpi should never be able to notice that I'm using more than one thread.

So my simplified code is this:

#include <cassert>
#include <thread>
#include "mpi.h"

void send(int rank, int& item)
{
   MPI::Request request = MPI::COMM_WORLD.Isend(&item, sizeof(int), MPI::BYTE, rank, 0);
   request.Wait();
}

void recv(int rank, int& item)
{
   MPI::COMM_WORLD.Recv(&item, sizeof(int), MPI::BYTE, rank, 0);
}

int main()
{
   MPI::Init();
   int ns[] = {-1, -1};
   int rank = MPI::COMM_WORLD.Get_rank();
   ns[rank] = rank;
   auto t_0 = std::thread(send, 1 - rank, std::ref(ns[rank])); // send rank to partner (i.e. 1 - rank)
   auto t_1 = std::thread(recv, 1 - rank, std::ref(ns[1 - rank])); // receive partner rank from partner
   t_0.join();
   t_1.join();
   assert( ns[0] == 0 );
   assert( ns[1] == 1 );
   MPI::Finalize();
}

Explanation of code: Two threads are executed on each processor. One tries to Isend some data to the partner and waits until this is done, the other one receives some data from the partner.

Question: Can I safely assume that most implementations of MPI don't choke up on this piece of code?

(Disclaimer: This piece of code is not designed to be exception-safe or particularly beautiful. It's for demo purposes only)

@HristoIliev Thanks for posting an answer there :) This is very good news (well, in part. That means there is a bug somewhere else -.-), because I don't have to throw away my concept _completely_ — stefan, Mar 04 '14 at 15:44

score 1 · Accepted Answer · answered Mar 04 '14 at 16:08

1

Question: Can I safely assume that most implementations of MPI don't choke up on this piece of code?

In practice - yes, if you add synchronisation (which your code is lacking); in theory - no. While it is possible that some implementations allow for serialised calls from different threads at the MPI_THREAD_SINGLE level (with Open MPI being such one - see here), the MPI standard requires that the library must be initialised at the MPI_THREAD_SERIALIZED level. If you intent for your software to be portable and to be able to compile and run correctly with other MPI implementations, you should not rely on some particular Open MPI behaviour.

That said, Open MPI can be configured to support multithreading (MPI_THREAD_MULTIPLE) when the library is built. The default is that MT support is not enabled for performance reasons. You can check the state of your particular installation using ompi_info:

$ ompi_info | grep MPI_THREAD_MULTIPLE
     Thread support: poxis (MPI_THREAD_MULTIPLE: no, progress: no)
                            ^^^^^^^^^^^^^^^^^^^^^^^

That particular build does not support multithreading and will always return MPI_THREAD_SINGLE in the provided output argument of MPI_Init_thread.

answered Mar 04 '14 at 16:08

Hristo Iliev

72,659
12
135
186

may I ask which parts of the code need to be synchronized? Does it suffice to wrap `Isend` and `Recv` with the same mutex? – stefan Mar 04 '14 at 16:11
All MPI calls must be synchronised with a critical section, no matter how it is implemented. – Hristo Iliev Mar 04 '14 at 16:13
Thanks, I think I now see a solution to my problem :) – stefan Mar 04 '14 at 16:16
Note that a single-line solution to your problem exists: `MPI_Allgather(MPI_IN_PLACE, 0, MPI_DATATYPE_NULL, ns, 1, MPI_INT, MPI_COMM_WORLD);` (whatever the C++ equivalent is). – Hristo Iliev Mar 04 '14 at 16:22
well this is a fairly simplified example code. In my real application, I need to collect results on the master nodes (coming in fairly randomly) and distribute jobs to the nodes. May I ask you to look at the following code and tell me if it's safe? http://pastie.org/8861734 – stefan Mar 04 '14 at 16:29
Honestly, C++ is not my language of choice, but if `std::lock_guard` works as a scoped lock, then it looks pretty safe. Also, if you store the requests in a simple array instead of `std::list`, you could use `MPI_Testany` instead of the loop. – Hristo Iliev Mar 04 '14 at 17:07
Indeed, std::lock_guard is a scoped lock. Thanks for your opinion, I feel much safer now :) – stefan Mar 04 '14 at 17:17

Safety guarantee for interleaved MPI Isend / Recv

1 Answers1