2

What libraries are available for parallel distributed cholesky decomposition of dense matrices in C/C++ in mpi environment?

I've found the ScaLAPACK library, and this might be the solution I'm looking for. It seems that it's a bit fiddly to call though, lots of Fortran <-> C conversions to do, which makes me think that maybe it is not widely used, and therefore maybe there are some other libraries that are used instead?

Alternatively, are there some wrappers for ScaLAPACK that make it relatively not too painful to use in a C or C++ environment, when one is already using MPI, and MPI has already been initialized in the program?

gsamaras
  • 71,951
  • 46
  • 188
  • 305
Hugh Perkins
  • 7,975
  • 7
  • 63
  • 71
  • 3
    Scalapack is the way to go in a distributed memory environment with distributed computations. If you can keep your matrices on one host and you don't use distributed matrix computations, lapack with openmp might be another possibility (e.g. you need N cholesky decomps, which can be distributed on N machines such that each machine can do cholesky with openmp on its m processors). And the memory alignment fiddling isn't that bad. I wrote some preproccessor directives which do the proper substitutions in the arguments. – Bort Jun 08 '13 at 14:35
  • Hi Bort. Ok, sounds good. So, Scalapack, since I only have a single matrix at a time, can't do task-level parallelism. What sort of scalability is scalapack capable of for Cholesky? 12 cores? 48 cores? 480 cores? I guess it depends somewhat on the size of the matrix? – Hugh Perkins Jun 09 '13 at 00:36
  • In terms of scalability there are more factors than just matrix size, e.g. inter node connectivity, cpu cache size, homogeneity of the network, network architecture and so on... If you have access to a big clusters and you need that number of cores, I would talk to the sysadmins and check what type of tests they have performed or let them perform the scalapack test routines to get an idea. – Bort Jun 09 '13 at 11:08
  • I suggest to read the header of [pdpbtrf.f](http://www.netlib.org/scalapack/double/pdpbtrf.f) for details and limitations on cholesky decomposition in scalapack. – Bort Jun 09 '13 at 11:17
  • @Bort, ok thanks. Cool! Question: how do people usually handle talking to fortran from C and/or C++? – Hugh Perkins Jun 09 '13 at 23:02
  • Edit: ok, I see. It seems calling fortran from c is not that hard in fact, it seems they use similar calling conventions, and you can pretty much call fortran directly from c, just adding an underscore, and making everything into a reference? – Hugh Perkins Jun 09 '13 at 23:40
  • Yes. Make sure you have all details about the data types. – Bort Jun 10 '13 at 10:13

2 Answers2

1

Are these dense or sparse matrices?

Trilinos is a huge library for parallel scientific computation. The sub-package Amesos can link to Scalapack for parallel, direct solution of dense systems and to UMFPACK, SuperLU or MUMPS for sparse systems. Trilinos is mostly in C++, but there are Python bindings if that's your taste. It might be overkill, but it'll get the job done.

toddwz
  • 521
  • 4
  • 9
Daniel Shapero
  • 1,869
  • 17
  • 30
  • Ah, good info thanks! And you're right, I should have specified sparse or dense. I've updated the original question to say that my matrices are dense. – Hugh Perkins Jun 10 '13 at 23:56
  • Yes, I tend to forget sparse matrices because I work with dense matrices all day long. Those libraries mentioned by korrok perform really well and should definitely mentioned as well. – Bort Jun 13 '13 at 19:58
1

Intel MKL might also be a choice, since it calls ScaLAPACK on the inside. Note that Intel supports student use of this library, but in this case you have to use an open source MPI version. Also the Intel forum is very helpful.

Elemental is also an option, written in C++, which is surely a big advantage when you want to integrate with your C/C++ application and the project leader, Jack Poulson is a very friendly and helps a lot.

OpenBLAS, SuperLU and PETSc are also interesting and you may want to read more in my answer.

Community
  • 1
  • 1
gsamaras
  • 71,951
  • 46
  • 188
  • 305