2

I am trying to parallelize a vertex-centred finite volume code on unstructured meshes. I first partition my mesh and then assign a processor to solve on each partition. Each processor/partition has vertices which it shares with one or more neighbouring partitions. As an example, consider the following where the mesh is partitioned and distributed amongst 6 processors:

enter image description here

At each of the shared vertices, data from the corresponding neighbouring processors need to be added together and sent back. For instance consider the partition handles by Proc 1. It shared vertices 'a' and 'b' with Proc 2, 'd' 'e' with Proc 4, and 'c' with Proc 2,4,5. Thus, in MPI terminology, Proc 1 will be involved in 3 separate AllReduce calls.

Presently, my code creates different MPI groups based on shared vertices. In the above example, Proc 1 is going to be a member of 3 such groups. For each group I create an MPI group communicator and carry out non-blocking all-reduces using MPI_IAllreduce. The algorithm works fine for a mesh with say 100 partitions. But the code crashes if I work with larger number of partitions. It seems MPI runs out of communicators to assign.

Could anyone suggest a possible workaround, or an alternative approach to the problem. While I can always work with smaller number of mesh-partitions, it would be nice to able to allow the code to work with as many partitions as possible (given the available resources).

rayd
  • 104
  • 8
  • Take a look at the [user guide of the PETSc library](http://www.mcs.anl.gov/petsc/petsc-current/docs/manual.pdf), in particular on section 2.4 Structured Grids Using Distributed Arrays, p49-50. The concept of ghost nodes is introduced : the values of these nodes are needed by the process, but the values are stored on other processes : communications are required. The function `DMDACreate2d()` can save you some time ! – francis Dec 19 '15 at 17:27
  • From what I understand, the DMDACreate2d() is useful for structured grids. I am working with unstructured grids, which need not have a nice block structure as shown in the above example. – rayd Dec 22 '15 at 07:51

0 Answers0