MPI_Isend: how to keep data safe until they're received?

Question

I'm to send many messages using MPI_Isend, but I'm not sure how to ensure my data are safe until they are received, and at the same time, do not use up all available mem.

Currently, I'm doing something like this:

vector<int*> outbox;
vector<MPI_Request> stats;

void run()
{
   while (!done()) 
   {
       //some magic
       //...

       sendMsg(rand(), getRecipRank());
  }
}

void sendMsg(int n, int recipRank)
{
    //Need to gen n random integers and send it to proc with rank recipRank
    //Can't block until the numbers are reveived. 
    //So how do I make sure these numbers are safe until then?


    int* data = (int*)malloc(sizeof(int) * n);
    //fill the data array with random numbers here

    MPI_Request req;
    MPI_Isend(data, n, MPI_INT, recipRank, 0, MPI_COMM_WORLD, &req);

    //Keeping the pointer to the array to free it later
    outbox.push_back(data); //Does this help keep the data safe until they're received?
    stats.push_back(req);
}

Then I have another function that occasionally goes through the stats vector to check the status of the send. If it's complete, then the function frees both the request and corresponding array in outbox.

I've tested this with a small number of messages, and it seems to work, but I'm not sure if it'll ALWAYS work or I'm just being lucky.

Nils_M · Accepted Answer · 2016-10-20T15:29:44.060

2

Looks good! If you allocate memory using malloc nobody will mess with it, but you. Since you don't mess with, it'll be safe.

This is a good pattern to interleave computation and communication by making use of more memory. If you want to put a limit on the memory usage you could impose a maximum length for the vector outbox and start using blocking sends when that length is reached.

Just to clarify: Your data is not "safer", because you push it into the vector outbox, it would be safe even without doing that. You push it into the vector to be able to free it later.

edited Oct 20 '16 at 15:29

answered Oct 20 '16 at 15:13

Nils_M

1,062
10
24

"Your data is not "safer..." Ah forgive my Java thinking. I was thinking because I kept a ref, it won't be garbage-collected. – 0x56794E Oct 20 '16 at 16:22
ah ok, I understand :) – Nils_M Oct 20 '16 at 16:27

score 0 · Answer 2 · answered Oct 20 '16 at 18:08

This works fine, but you even don't need to resort to C-malloc.

You can safely use C++ constructs like:

new[] pointers in a std::vector<int*>
std::vector<std::unique_ptr<int[]>> (I'd prefer that in your situation)
std::vector::data() of vectors in vectors. For that however ensure to not call any non-const methods on the vectors that provide the memory called until the request is completed.

It would not be safe to use std::vector<int[10]> outbox or std::vector<std::array<int, 10>> outbox as this memory would be refer to directly inside the outbox vector, which may get reallocated (resized) on further calls to push_back. But that only matters for compile-time-known n. std::vector<std::unique_ptr<std::array<int, 10>>> would be fine again.

MPI_Isend: how to keep data safe until they're received?

2 Answers2