2

I'm trying to gather 2 matrices computed on different processes, i'm using the mpi allgatherv function to do so but I'm surprised by the results.

In the example I'm working on, the shape of the matrices is (20,80) and I would like to gather them into one matrix with (40,80) shape; with the matrix computed by process 0 stored in the 20 first rows.

Here is a sample of the code I'm using (nbRegimes is 0 in the example):

boost::mpi::communicator world;
int  rank = world.rank();

std::vector< ArrayXXd> phiOutLoc(nbRegimes);

for (int iReg = 0; iReg < nbRegimes; ++iReg)
    phiOutLoc[iReg].resize(nScenarioCur * nDimension, nbPoints);
    
... computation and storage into phiOutLoc[iReg] ...
    
//gathering results
ArrayXXd storeGlob(nScenarios * nDimension, nbPoints);
for (int iReg = 0; iReg < nbRegimes; ++iReg)
{
    boost::mpi::all_gatherv<double>(world, phiOutLoc[iReg].data(), phiOutLoc[iReg].size(), storeGlob.data());   
}

For instance, here is the beginning of the first row of phiOutLoc[iReg] computed on process 0 and process 1 :

rank 0

0      -353509      -366699      -379888      -393077      -406267      -419456      -432646  ... 

rank 1

0      -399021      -413908      -428795      -443683      -458570      -473457      -488345  ...

Those rows should be stored respectively in the index 0 row of storeGlob and in the index 20 row; if I understood the all_gatherv function behaviour correctly.

Howewer the index 0 row of storeGlob looks like this :

storeGlob row 0:
 0      -366699      -393077      -419456     ...  0      -413908      -443683      -473457

and the index 20 row of storeGlob :

storeGlob row 20:
-353509      -379888      -406267      -432646 ... -399021      -428795      -458570      -488345

The index 0 row of storeGlob is filled with the even indices of the first rows of phiOutLoc[iReg] matrices. The odd indices are stored in the index 20 row.

I can't really understand why the gathering is done that way.

Is this behaviour normal and is there a way to gather the two matrices the way I would like?

Daniel Langr
  • 22,196
  • 3
  • 50
  • 93
Shinra_SGr
  • 21
  • 1
  • Why are you using `all_gatherv` when all the processes send the same amount of data? Morevoer, I don't see any `all_gatherv` in [Boost.MPI headers](https://github.com/boostorg/mpi/blob/develop/include/boost/mpi/collectives.hpp). Also, if each process "sends" `nScenarioCur * nDimension * nbPoints` numbers, shouldn't the output array be of `nScenarioCur * nDimension * nbPoints * world.size()` size? – Daniel Langr Jun 26 '20 at 14:52
  • As for interleaving, consider this quote from the [Eigen documentation](https://eigen.tuxfamily.org/dox/group__TopicStorageOrders.html): _"If the storage order is not specified, then Eigen defaults to storing the entry in **column-major**."_ I think you should use a row-major storage order if you want to gather data with MPI and preserve the number of columns. – Daniel Langr Jun 26 '20 at 15:01
  • Thanks for your comments. I'm using all_gatherv because there are cases where I won't have the same amout of data. Regarding the size of the output, you're right but the way I did it is equivalent. Finally? I checked the rowMajor storage option for eigen array and it was indeed my problem. Thank you – Shinra_SGr Jun 26 '20 at 16:08

0 Answers0