0

I'm trying to pass a diagonal matrix W to an Rcpp function. The problem is that W has size 1,000,000 by 1,000,000, which is (I think) well outside the limits Armadillo allows (even when using a C++11 compiler with ARMA_64BIT_WORD enabled).

Since W is a diagonal matrix, it is extremely sparse. For that reason, I first generated a dense representation of W (using the Matrix package function Diagonal). I then passed these compressed representation of W to my function. I thought this would solve any memory problems. Here's a small example:

C++ Code:

#define ARMA_64BIT_WORD 1
#include <RcppArmadillo.h>
// [[Rcpp::depends(RcppArmadillo)]]
// [[Rcpp::plugins(cpp11)]]

using namespace Rcpp;
using namespace arma;


// [[Rcpp::export]]
int test(sp_mat W){
  
  return 0;
  
}

R Code:

# define the diagonal matrix
nrows <- 1e6
W <- Matrix::Diagonal(nrows)

# call Rcpp function
test(W)

However, despite using a compressed representation, I still get this error.

error: SpMat::init(): requested size is too large; suggest to compile in C++11 mode and/or enable ARMA_64BIT_WORD

Is there any way to handle W so that I can pass it into test and perform matrix operations?

A similar question was asked here. However, I think the solution provided by @Dirk worked in that case since the input matrix was still small compared to my W.

ralph
  • 223
  • 1
  • 4
  • _If_ the matrix has sufficient sparsity _and_ you use the right tools you can do this. You need to experiment. What $N$ works is also a function ... of your machine and its RAM allotment. There are different ways to construct sparse matrices. We have some converters from R thanks the Google Summer of Code project a few years ago; you can also try directly in C++ using Armadillo functions to triangulate the issue. – Dirk Eddelbuettel Oct 19 '20 at 13:59
  • BTW your example works on my machine just fine--even for 1e7 and 1e8. Maybe I have more RAM than you---though it is not "that much". – Dirk Eddelbuettel Oct 19 '20 at 16:09
  • Interesting, thanks for testing it. I have 16GB of RAM on my machine. Is there any way I can tell if a C++11 compiler is *definitely* being used and ARMA_64BIT_WORD is enabled? I removed the comments from the config.hpp file, and I still get the above error. – ralph Oct 19 '20 at 20:07
  • 1
    Maybe I should move that to an answer instead of a comment... Do `sourceCpp(filename, verbose=TRUE)` and you should see the C++ standard invoked---R 4.0.* itself makes it C++11 these days. What does `g++ --version` say for you? – Dirk Eddelbuettel Oct 19 '20 at 20:31
  • It says `g++ -std=gnu++11`. It also has `C:/rtools40/mingw32/bin/g++` listed in the output. Could this mean I'm not using the 64 bit version? There's also this call to `dyn.load`: `.sourceCpp_1_DLLInfo <- dyn.load('C:/Users/ralph/AppData/Local/Temp/RtmpaKLEIw/sourceCpp-i386-w64-mingw32-1.0.5/sourcecpp_9bc23c31b/sourceCpp_9.dll')` – ralph Oct 19 '20 at 20:47
  • With RTool 4.0 you should be fine, and also be using the 64-bit version by defaiult. What I can't remember is whether Windos gets in your way or not. But in any event... no Rcpp or RcppArmadillo issue here AFAICT. – Dirk Eddelbuettel Oct 19 '20 at 20:57
  • Thanks for the help so far. I added `CXX_STD =-CX11` and `PKG_CXXFLAGS=-ARMA_64BIT_WORD` as environment variables. Now I get the error: `g++.exe: error: ARMA_64BIT_WORD: No such file or directory` – ralph Oct 19 '20 at 21:23
  • You need a `D` for `define` there: `-DARMA_64BIT_WORD`. And no minus before `CX11` instead `CXX_STD = CXX11`. – Dirk Eddelbuettel Oct 19 '20 at 21:29
  • Strange. I can see -DARMA_64BIT_WORD in the cmd line call, but still the same error. `C:/Rtools/mingw_32/bin/g++ -std=gnu++11 -I"C:/PROGRA~1/R/R-35~1.0/include" -DNDEBUG -I../inst/include -fopenmp -I"C:/Users/ralph/Documents/R/win-library/3.5/Rcpp/include" -I"C:/Users/ralph/Documents/R/win-library/3.5/RcppArmadillo/include" -I"C:/Users/ralph/Desktop/asp" -DARMA_64BIT_WORD -O2 -Wall -mtune=generic -c maincpp.cpp -o maincpp.o` I will keep at it. – ralph Oct 19 '20 at 22:09
  • (If it is in your source file as originally then you don't have the problem as you don't need to set it again. Ditto with C++11 which is already the default.) – Dirk Eddelbuettel Oct 19 '20 at 22:12
  • I changed to a different windows machine and it ran fine, even for 1e7 and 1e8. I might try a fresh installation of R and Rtools on the original machine and try again. Thanks for the assistance @Dirk. – ralph Oct 20 '20 at 10:43
  • Glad you got it sorted out. Let me put closure on it by posted my (slightlty edited / simplified) version of your code. Feel to accept it, it shows another trick or two. But by and large you had this right already... – Dirk Eddelbuettel Oct 20 '20 at 13:39

1 Answers1

1

The originally posted code was actually correct, but somehow the machine it was running on was not---see the discussion above for details.

As an add-on, here is a slightly edited version of the code without global namespace, actual access to the matrix (passing was enough, this is simply more explicit) and returning void. We also added the usual 'call it from R for me' trick one can do with Rcpp Attributes.

Code

// lightly edited version of question, working fine for me
#define ARMA_64BIT_WORD 1
#include <RcppArmadillo.h>
// [[Rcpp::depends(RcppArmadillo)]]

// [[Rcpp::export]]
void spmatTest(arma::sp_mat M) {
  // show a simple access of matrix,
  // here we just sum elements on diagonal
  Rcpp::Rcout << "Sum of diag is "
              << arma::as_scalar(arma::sum(M.diag()))
              << std::endl;
}

/*** R
spmatTest( Matrix::Diagonal( 1e6 ))
spmatTest( Matrix::Diagonal( 1e7 ))
spmatTest( Matrix::Diagonal( 1e8 ))
*/

Output

R> Rcpp::sourceCpp("~/git/stackoverflow/64428776/answer.cpp")

R> spmatTest( Matrix::Diagonal( 1e6 ))
Sum of diag is 1e+06

R> spmatTest( Matrix::Diagonal( 1e7 ))
Sum of diag is 1e+07

R> spmatTest( Matrix::Diagonal( 1e8 ))
Sum of diag is 1e+08
R>
Dirk Eddelbuettel
  • 360,940
  • 56
  • 644
  • 725