0

I'm working on a MPI C program that compute a relaxed state of a hotplate by averaging it's neighbors. So, edge's of each processor need the values from the neighboring proccessor. The problem is fairly basic, but I'm struggling to efficiently use the MPI_SendRecv in two dimensions.

I currently have a 2D grid of processors and each processor has it's own grid of 2D floats of values. After each averaging, processors have to send their edge values (I will be vary the number of rows/cols to send to find optimal sizes given the bandwidth/processing power of different setups once it's all working) to the processor next to him. I've been trying to do all my homework on how to send and recieve, but hit a snag.

MPI_SendRecv does best with contiguous blocks of memory, so I can have each processor allocate their needed size (+padding to store neighboring values). If I broke my hotplate up into meta-rows and passed the last three rows to the neighbor, it would be easy. But my problem has processors not just above and below, but left and right in the 2D array.

So far, the best way I could come up with is to copy the right (x) number of columns into a new contiguous block of memory and then ship that contiguous block to the processor to my right and have that processor copy those values into the 2d array that it has, but this seems super inefficient in my mind. The second option I could think of was to have a second block with the same values but organized by columns instead of rows, but any write would require writing to two places.

Is there a more efficient way? Does this problem have a name that I could research more on? Are there any MPI ways to do this that I haven't found yet?

Thanks!

Jacob Holloway
  • 887
  • 8
  • 24
  • 3
    This is a fairly basic problem in scientific parallel computing and one of the parallel "design patterns". Search for "halo exchange in 2D" or "ghost cells exchange in 2D". There are posts concerning this here, so your question is most likely a duplicate. In any case, the column exchange in C is performed by constructing a vector datatype (using `MPI_Type_vector`) with stride equal to the local subarray width. – Hristo Iliev Oct 07 '15 at 20:47
  • Ok, maybe not really that many questions concerning 2D halo exchange here. But you should still be able to find many examples and MPI tutorials using your favourite search engine and the keywords from my previous comment. – Hristo Iliev Oct 07 '15 at 20:52
  • Thanks! Just knowing how to search for answers is enough for me! (I only post when I hit a dead end...). I'll look into that for sure and try to find the duplicate. :) – Jacob Holloway Oct 07 '15 at 21:02
  • Try [that one](http://stackoverflow.com/questions/6598578/most-appropriate-mpi-datatype-for-block-decomposition). The images in the question are missing, but the answer contains some very useful information (Jonathan Dursi, as usual). I will fix the images and post a full 2D halo exchange example... some day. – Hristo Iliev Oct 07 '15 at 21:06
  • 1
    Take a look at the `DMDA` structure of the Petsc library. My guess is that it is doing exactly what you are looking for. See [`DMDACreate2d()`](http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DM/DMDACreate2d.html). Edges are called ghost points. See [`DMDAVecGetArray()](http://www.mcs.anl.gov/petsc/petsc-current/docs/manualpages/DM/DMDAVecGetArray.html) and the [manual of petsc](http://www.mcs.anl.gov/petsc/petsc-current/docs/manual.pdf) on page 50+. – francis Oct 09 '15 at 19:16
  • @francis, I'll have to take a look at that sometime next week when I have more free time. :) Thank. – Jacob Holloway Oct 09 '15 at 19:27

0 Answers0