1

Let's say you have a 3d array A[I][J][K], but you want to permute it into B[J][K][I].

This problem is similar to, but different from, the 2-d array transpose problem discussed here: Can you transpose array when sending using MPI_Type_create_subarray?.

The MPI standard provides examples of multi-dimensional array operations with user-defined datatypes, but not this particular case: http://www.mpi-forum.org/docs/mpi-11-html/node61.html

Community
  • 1
  • 1
Rob Latham
  • 5,085
  • 3
  • 27
  • 44

1 Answers1

1

The way to do this is with MPI_TYPE_VECTOR and MPI_TYPE_CREATE_HVECTOR, but the devil is in the details. 

/* data[I][J][K] is I by J by K (stored in array 'dim_sizes[] = {I, J, K}'
   and we want permuted[J][K][I] */

/* new innermost dimension is I items, strided across the old JK face*/
MPI_Type_vector(dim_sizes[0], 1, dim_sizes[1]*dim_sizes[2], 
    MPI_DOUBLE, &one_d);
MPI_Type_commit(&one_d);

/* new middle dimenson is K items, strided over the K row, which isn't
 * actually a stride in this case.  We use hvector here because we 
 * operate directly in terms of array items */
MPI_Type_create_hvector(dim_sizes[2], 1, sizeof(double), one_d, &two_d);
MPI_Type_commit(&two_d);

/* new outermost dimension is J items, strided over the old J row */
MPI_Type_create_hvector(dim_sizes[1], 1, dim_sizes[2]*sizeof(double), two_d,
    &transposed_type);
MPI_Type_commit(&transposed_type);

Now you can feed transposed_type to your send/receive call or make it your MPI file view.

Hristo Iliev
  • 72,659
  • 12
  • 135
  • 186
Rob Latham
  • 5,085
  • 3
  • 27
  • 44
  • 1
    You have to call `MPI_Type_commit` for each datatype that was created. `MPI_Type_hvector` is long deprecated - one should use `MPI_Type_create_hvector` instead. I've edited your answer accordingly. – Hristo Iliev Oct 28 '12 at 11:50
  • good catch on the type_commit. My personal preference is to commit the final type and not the constituent types, since one_d and two_d will never be used in a send/receive or file_set_view call. Thank you also for updating the deprecated type_hvector call. – Rob Latham Oct 29 '12 at 19:34
  • You're absolutely right - there is no need to commit types which are not used in communication calls, but it makes a good programming practice to do so, since later one might decide to use one of the intermediate types. – Hristo Iliev Oct 29 '12 at 20:33