Send and Receive derived Datatypes with MPI

Question

I want to send in the most easy way the anti-diagonal elements of a NXN matrix from the root Process to the other process. Sadly at the moment i can not test my codes because the compute nodes are down. Can someone check my simple code?

I am not sure if i send the anti-diag elements of A correctly. And will the anti-diagonal elements land in the receive buffer B?

#include "mpi.h"
#include <stdio.h>

int main(int argc, char** argv){

MPI_Init(&argc,&argv);
int size, rank;

MPI_Comm_rank(MPI_COMM_WORLD,&rank);
MPI_Comm_size(MPI_COMM_WORLD,&size);

int N=4;
double A[N][N];
double B[N];
MPI_Datatype antidiag;

int* blockleng=(int*)malloc(N*sizeof(int));
int* displace=(int*)malloc(N*sizeof(int));

for(int i=0; i<N; ++i){
    blockleng[i]=1;
    displace[i] = (i+1)*(N-1);
}

MPI_Type_indexed(N,blockleng,displace, MPI_DOUBLE,antidiag);
MPI_Type_commit(&antidiag);
MPI_Status status;

if(rank==0){
    A= {
            {1.0,5.0,9.0,13.0},
            {2.0,6.0,10.5,14.5},
            {3.0,7.2,11.0,15.0},
            {4.0,8.0,12.0,16.0}
    };


    MPI_Send(A,1,antidiag,1,100,MPI_COMM_WORLD);

}
if(rank==1){
    MPI_Recv(B,1,antidiag,0,100,MPI_COMM_WORLD,status);
}

MPI_Type_free(&antidiag);

MPI_Finalize();
return 0;
}

Answer 1

You have a couple issues in your program. Rather than posting the whole program I will focus on the issues.

1. The declaration of variable inside the loop holds from C99 on. In case you need to use the older standard this is how the loop should look like,

   int i, j;  
   for(i=0; i<N; ++i){
       blockleng[i]=1;
       displace[i] = (i+1)*(N-1);
   }
   

2. When creating an MPI datatype with MPI_Type_indexed your last argument, the new type, has to be passed as a pointer/handle:

   MPI_Type_indexed(N, blockleng, displace, MPI_DOUBLE, &antidiag);
   

3. The way you assign the values to matrix A is possible only as part of initialization combined with the declaration, more here. You can do that, of course, separately for rank 0 and other ranks, but you can also read the matrix data from a file only on rank 0,

   char* file_in = "matrix_A.txt";
   FILE *fpi;
   
   fpi=fopen(file_in, "r");
   for (i=0; i<N; i++)
       for (j=0; j<N; j++)
           if (!fscanf(fpi, "%lf", &A[i][j]))
               break;
   

   As part of the if(rank==0){ block. You can also make the matrix filename a command line argument which will give you even more flexibility. At that point you'd also want to have N as a command line argument to be able to define the size of the matrix your are reading. The file used in this example had a simple structure,

   1.0 5.0 9.0 13.0
   2.0 6.0 10.5 14.5
   3.0 7.2 11.0 15.0
   4.0 8.0 12.0 16.0
   

4. The send part in the same block sends only to a single rank, rank 1. If I understood your question correctly you want to send the antidiagonal to all other ranks from rank 0 which requires the following loop,

   for (i=1; i<size; i++)
       MPI_Send(A, 1, antidiag, i, 100, MPI_COMM_WORLD);
   

5. Similarly, the receive part should hold for all ranks other than 0,

   else{ 
       MPI_Recv(B, N, MPI_DOUBLE, 0, 100, MPI_COMM_WORLD, &status);
   }
   

   Note that there are other corrections here - you are receiving your data in an N sized buffer/array B, not the NxN matrix A. atidiag is a type created for a matrix A (or a N*N element 1D buffer, it's equivalent in this case) - it does not hold for N element buffer B. Thus you need to change the MPI_Recv to expect/retrieve N elements of type MPI_DOUBLE instead. This is a neat feature as it allows you to receive your data into arrays of different structure than the one sending. You also need to pass a pointer to status, hence &status.

Finally, you can print your results,

 if (rank != 0){
    printf("Rank %d:\n", rank);
    for (i=0; i<N; i++)
      printf("%.2lf ", B[i]);
  }
  printf("\n\n");

while keeping in mind that MPI printing to stout is not ordered based on rank numbers and more so, ranks may interrupt each other during printing. Flushing stdout may help, but better option is to write to a file. In this case, on my system, the output was printed just as it should be. Tested with up to 6 processes.

Last note - normally in case of 1 process sending to all others you should consider using collective communication functions like MPI_Bcast. These functions are highly optimized and outperform Send\Recv for cases they are specialized in. Now, with Bcast you would need to actually store the received antidiagonal in the A matrix on all ranks, and then copy it to the array/buffer B on ranks other than 0. This would add a couple steps, so I left it as it is in your post.