2

I need to create an MPI derived type to represent a class in my program. The class is fairly straight forward, but large (about 75 data members**). All the data members are single values, 1D arrays, or 2D arrays. Here is an example:

class RestartData {
  int dsr;
  double firea2sorgn;
  int ifwoody[NUM_PFT];
  double rootfrac[MAX_ROT_LAY][NUM_PFT];
  ....
  ....
}

I think that using the MPI_Type_struct is appropriate. (e.g. http://www.open-mpi.org/doc/v1.5/man3/MPI_Type_struct.3.php)

And I more or less follow the example in this question: struct serialization in C and transfer over MPI, but I am not sure how to handle the 2D arrays. Can I make an MPI_Type_struct that contains several MPI_Type_vectors? I have been unable to find an example of creating an MPI_Type_struct containing 2D arrays. Am I on the right approach?

Thanks in advance.

** I think I understand the possible problems with passing a single large message, but in this case, the message is passed infrequently, and at a natural synchronization point (slaves sending data back to the master when they are done crunching numbers)

Community
  • 1
  • 1
tbc
  • 1,679
  • 3
  • 21
  • 28

2 Answers2

3

Derived types in MPI can be freely constructed from other derived types and then used to further create other derived types.

2D arrays, as long as they are contiguous in memory as in your case, are not that different from 1D arrays. When it comes to the rootfrac member, you could either create a contiguous datatype with MAX_ROOT_LAY * NUM_PFT elements of type MPI_DOUBLE or you could create a contiguous datatype (let's call it t_dbl_pft) with NUM_PFT elements of type MPI_DOUBLE and then use it to create another contiguous datatype with MAX_ROOT_LAY elements of type t_dbl_pft. Another option is to not create a datatype at all since the MPI structured type constructor takes a separate block length (i.e. number of elements) for each element of the structure.

For example, the following type describes the data members that you've shown:

#include <cstddef> // for offsetof

MPI_Type t_1d;
MPI_Type_contiguous(NUM_PFT, MPI_DOUBLE, &t_1d);
// No need to commit - this type is not used for communication

MPI_Type t_restart;
int counts[4] = { 1, 1, NUM_PFT, MAX_ROT_LAY };
MPI_Type types[4] = { MPI_INT, MPI_DOUBLE, MPI_INT, t_1d };
MPI_Aint displs[4] = {
   offsetof(RestartData, dsr),
   offsetof(RestartData, firea2sorgn),
   offsetof(RestartData, ifwoody),
   offsetof(RestartData, rootfrac),
};
MPI_Type_create_struct(4, counts, displs, types, &t_restart);
MPI_Type_commit(&t_restart);

// The 1D type is no longer needed
MPI_Type_free(&t_1d);

Note that you have to either create the MPI datatype inside a member function of RestartData or declare the routine where the type is created as friend to the class so that it could access the private and protected data members. Also note that offsetof only works with POD (plain old data) classes, e.g. no fancy constructors and members of types like std::string.

Hristo Iliev
  • 72,659
  • 12
  • 135
  • 186
  • +1: I'm with Hristo on this one, preferring strong typing (well, as near as C+++MPI can manage) over weak typing (*it's all bytes or chars*) as proposed by @Adam's answer. – High Performance Mark Sep 25 '13 at 09:44
  • For curiosity (and completeness) how would you handle a member of non POD type? – tbc Sep 25 '13 at 16:04
  • One usually uses `MPI_Pack` and `MPI_Unpack` to implement portable manual (de-)serialisation into an intermediate buffer that is afterwards sent/received using the `MPI_PACKED` datatype. – Hristo Iliev Sep 25 '13 at 20:30
  • update - it ended up being fairly easy to get a prototype working using Hristo's first example: "create a contiguous datatype with MAX_ROOT_LAY * NUM_PFT elements of type MPI_DOUBLE...". Seems great for some initial testing, but feels like it might be tedious to maintain? – tbc Oct 01 '13 at 15:57
2

The easiest way to go is to just treat the whole object as a big buffer:

MPI_Datatype datatype;

MPI_Type_contiguous(sizeof(RestartData), MPI_BYTE, &datatype);
MPI_Type_commit(&datatype);

I don't see the benefit of telling MPI about the internal structure of your class.

Alternatively a 2D array is an array of 1D arrays. So (I imagine) you could use one call to MPI_Type_contiguous per array dimension to build up the array datatype.

Adam
  • 16,808
  • 7
  • 52
  • 98
  • Hmm, that definitely looks like an easy way to get started. I will try that and see what happens. Thanks. – tbc Sep 25 '13 at 01:51
  • This is a terrible answer. Why use MPI then? He could just use 0MQ or even sockets to pass messages around. – Hristo Iliev Sep 25 '13 at 08:43
  • @HristoIliev what is there to gain from specifying the struct? – Adam Sep 25 '13 at 08:48
  • MPI is all about source code portability. For example the program could be run on a heterogeneous cluster with mixed 32/64-bit nodes or mixed little/big-endian nodes. Also MPI is allowed to perform conversions on `MPI_CHAR` and one should use `MPI_BYTE` instead if conversion is to be suppressed. – Hristo Iliev Sep 25 '13 at 09:10
  • Source code portability between different systems, not within a single system. I can't even fathom how to start a job with different binaries on different nodes. If the same binary handles memory differently on different nodes then there's bigger problems. Good point about the `MPI_BYTE`. – Adam Sep 25 '13 at 09:26
  • The only benefit I see from a non-`MPI_BYTE[]` datatype is when you use the built-in aggregation functions for POD types. I.e. a MPI_Reduce. Those won't work on custom datatypes anyway. All the rest like sends, broadcasts, scatter/gather, etc, practically only work on buffers anyway. What difference does it make if it's a `MPI_DOUBLE` or `MPI_BYTE[sizeof(double)]`? – Adam Sep 25 '13 at 09:31
  • Heterogeneous architectures are more common that you might think, e.g. Intel Xeon Phi boards support MPI message passing to and fro their host systems. In that case structured types facilitate passing data with different member alignment (and data representation conversion where necessary). – Hristo Iliev Sep 25 '13 at 10:05