2

I am thinking about a problem that requires sharing an array as follows:

Suppose int array[10] and there are 3 processes so that;

process 0 gets array[0:3] 3 is included.
process 1 gets array[3:6] 6 is included.
process 2 gets array[6:9] 9 is included.

However I am not sure how to split this array between processes like this. To be more specific, this array is for Sparse Matrix in CSR format, and array represents rows.

How can I figure this out in MPI C/C++.

HARUN SASMAZ
  • 567
  • 1
  • 7
  • 17
  • 1
    `MPI_Scatterv()` – Gilles Gouaillardet May 19 '20 at 13:15
  • @GillesGouaillardet can you elaborate how to use that? I know it puts strides between elements but I am not sure how to use that in this concept. – HARUN SASMAZ May 19 '20 at 13:52
  • @GillesGouaillardet, according to the documentation, `MPI_Scatterv` cannot be used to scatter overlapping ranges. – Evg May 19 '20 at 13:55
  • Do I understand it correctly that you want to scatter overlapping ranges? – Evg May 19 '20 at 13:58
  • @Evg actually when there are 9 rows and 3 processes, each process get 3 rows. However, we add one additional element so that each process knows where to stop. That is why there are 10 elements in the array and each process gets 4 elements. – HARUN SASMAZ May 19 '20 at 14:33
  • `MPI_Scatterv` is not guaranteed to work with overlapping ranges (though it [may work in practice](https://stackoverflow.com/questions/36582696/mpi-python-scatterv-and-overlapping-data) - this requirement has its root in `MPI_Gatherv` where it makes perfect sense). You can manually send each subrange to each process. It might be not optimal but is guaranteed to work. – Evg May 19 '20 at 14:37
  • 1
    So `MPI_Scatterv()` has to be invoked twice in order to achieve the expected result while respecting the MPI standard :-) – Gilles Gouaillardet May 19 '20 at 14:48
  • @Evg you mean sending ranges through MPI_Send from Master thread? In that example, it seems working but I am confused about the role of displs. Is it for the starting index? or stride? – HARUN SASMAZ May 19 '20 at 16:18
  • @GillesGouaillardet can you provide a pseducode or a decription about how I can achieve my goal with two Scatterv? – HARUN SASMAZ May 19 '20 at 16:19

1 Answers1

4

A non-standard-conforming implementation using MPI_Scatterv:

int arr[10];
const int counts[] = {4, 4, 4};
const int displs[] = {0, 3, 6};

int recv_buff[4];
MPI_Scatterv(arr, counts, displs, MPI_INT, recv_buff, 
    4, MPI_INT, 0, MPI_COMM_WORLD);

displs are just simple offsets and are specified as follows:

Original array arr[10]:
[ 0 1 2 3 4 5 6 7 8 9 ]
  ^ displs[0]
        ^ displs[1]
              ^ displs[2]

This is not guaranteed to work because subarrays overlap:

The specification of counts, types, and displacements should not cause any location on the root to be read more than once.

As Gilles Gouaillardet noted in comments, to avoid overlaps you can call MPI_Scatterv twice:

int arr[10];
const int counts1[] = {4, 0, 4};
const int counts2[] = {0, 4, 0};
const int displs[]  = {0, 3, 6};

int recv_buff[4];
MPI_Scatterv(arr, counts1, displs, MPI_INT, recv_buff, 
    counts1[rank], MPI_INT, 0, MPI_COMM_WORLD);
MPI_Scatterv(arr, counts2, displs, MPI_INT, recv_buff, 
    counts2[rank], MPI_INT, 0, MPI_COMM_WORLD);

Alternatively, you can use plain MPI_Sends/MPI_Gets.

Evg
  • 25,259
  • 5
  • 41
  • 83
  • So, displs is basically the starting point of each rank? Btw, I like this approach. – HARUN SASMAZ May 19 '20 at 17:04
  • @HarunSasmaz Yes, process `i` receives the subarray that starts at `arr + displs[i]` and has length of `counts[i]`. – Evg May 19 '20 at 17:10