MPI Receive and Send for Eigen compressed sparse matrix

Question

What is the best (fastest) way to send and receive sparse matrix that is defined using Eigen library?

Currently, I am creating a value, row and column vector (from a pre-defined sparse matrix) using only the nonzero elements and sending/receiving these 3 vectors one by one. These vectors are simple std::vector

if (0 == myrank) {
    Sz.insert(0,0) = 0.5;       Sz.insert(1,1) = -0.5;

    //------ preparing to send ----------
    shape[0] = Sz.rows();
    shape[1] = Sz.cols();
    int size=Sz.nonZeros();
    val.resize(size); inner.resize(size); outer.resize(size);
    cout << "val-size = "<< val.size() << endl;
    int counter=0;
    for (int k=0; k<Sz.outerSize(); ++k) {
        for (CrsMatrixType::InnerIterator it(Sz,k); it; ++it)  {
            val[counter]=it.value();
            inner[counter]=it.col();
            outer[counter]=it.row();
            counter++;
        }
    }
    assert(counter==size);
    MPI_Send(&shape[0],2,MPI_INT,1, 100, MPI_COMM_WORLD);
    MPI_Send(&size,1,MPI_INT,1, 101, MPI_COMM_WORLD);
    MPI_Send(&val[0],size,MPI_DOUBLE,1, 102, MPI_COMM_WORLD);
    MPI_Send(&inner[0],size,MPI_INT,1, 103, MPI_COMM_WORLD);
    MPI_Send(&outer[0],size,MPI_INT,1, 104, MPI_COMM_WORLD);


}

Later, I receive them using

if (1 == myrank) {

    //------ preparing to receive ----------
    shape.resize(2);
    int size;
    MPI_Recv(&shape[0],2,MPI_INT,0, 100, MPI_COMM_WORLD, &status);
    MPI_Recv(&size,1,MPI_INT,0, 101, MPI_COMM_WORLD, &status);

    val.resize(size); inner.resize(size); outer.resize(size);
    MPI_Recv(&val[0],size,MPI_DOUBLE,0, 102, MPI_COMM_WORLD, &status);
    MPI_Recv(&inner[0],size,MPI_INT,0, 103, MPI_COMM_WORLD, &status);
    MPI_Recv(&outer[0],size,MPI_INT,0, 104, MPI_COMM_WORLD, &status);


    Sz.resize(shape[0],shape[1]);
    Sz.reserve(size); // allocate room for nonzero elements only.
    for (int k=0; k<Sz.outerSize(); ++k) {
        Sz.coeffRef(inner[k],outer[k]) = val[k];
    }

    cout << "my RANK " << myrank << endl;
    cout << Sz << endl;
}

and add them to the sparse matrix of rank 1.

Is there any better way to do this? Thanks.

First, you should send `shape` and `size` in one shot. Then you can create a `struct` with two `MPI_INT` and one `MPI_ DOUBLE` that describes an element of the matrix, allocate and fill it, build a derived datatype for that, and the send `size` elements in one shot. — Gilles Gouaillardet, Apr 22 '18 at 00:45
Thanks! :D I went with @chtz comment below. It avoids making temporaries and directly fill the matrix in rank 1. — qubit10, Apr 22 '18 at 16:51

score 3 · Answer 1 · answered Apr 22 '18 at 00:19

I generally suggest to use the compressed form to transfer sparse matrices. And there is no need to copy values into temporary std::vectors.

void sendSparseEigen(const Ref<const SparseMatrix<double>,StandardCompressedFormat>& mat) {
    int rows=mat.rows, cols=mat.cols, nnz=mat.nonZeros();
    assert(rows==mat.innerSize() && cols==mat.outerSize());
    assert(mat.outerIndexPtr()[cols]==nnz);
    int shape[3] = {rows, cols, nnz};
    MPI_Send(shape              ,3   ,MPI_INT,   1, 100, MPI_COMM_WORLD);
    MPI_Send(mat.valuePtr()     ,nnz ,MPI_DOUBLE,1, 101, MPI_COMM_WORLD);
    MPI_Send(mat.innerIndexPtr(),nnz ,MPI_INT,   1, 102, MPI_COMM_WORLD);
    MPI_Send(mat.outerIndexPtr(),cols,MPI_INT,   1, 103, MPI_COMM_WORLD);
}

void receiveSparseEigen(SparseMatrix<double> &out){
    int shape[3];
    MPI_Recv(shape,3,MPI_INT,0, 100, MPI_COMM_WORLD, &status);
    int rows=shape[0], cols=shape[1], nnz=shape[2];
    out.resize(rows, cols);
    out.reserve(nnz);
    MPI_Recv(out.valuePtr(),  nnz, MPI_DOUBLE,0, 101, MPI_COMM_WORLD, &status);
    MPI_Recv(out.innerIndexPtr(),nnz, MPI_INT,0, 102, MPI_COMM_WORLD, &status);
    MPI_Recv(out.outerIndexPtr(),cols,MPI_INT,0, 103, MPI_COMM_WORLD, &status);
    out.outerIndexPtr()[cols] = nnz;
}

Disclaimer: I'm no MPI expert, I copied all MPI related code from your example -- obviously you should handle all possible errors generated by MPI somehow. And the code above is untested.

Works like charm, although I have not test the speed/memory explicitly. It is clear that your solution avoids creating temporary that saves memory and redundant operations. Thanks! One more thing, in future, if someone else is using this solution, make sure to compress the sparse matrix before you pass. mat.makeCompressed(); — qubit10, Apr 22 '18 at 17:00
True, you could actually remove the `StandardCompressedFormat` flag from the `Ref` and just `assert(mat.isCompressed());` Or pass a non-const `SparseMatrix&` and call `makeCompressed()` inside the first function. — chtz, Apr 23 '18 at 07:38

MPI Receive and Send for Eigen compressed sparse matrix

1 Answers1