Stack Overflow
Stack Overflow

Questions tagged [r-bigmemory]

r-bigmemory is an R package (not to be confused with bigmemory referring to Java ehcache) refers to the suite of bigmemory packages (bigmemory, bigalgebra, bigtabulate) that allow for creating, storing, accessing, and manipulating massive matrices. Matrices are allocated to shared memory and may use memory-mapped files.

r-bigmemory is an R package to create, store, access, and manipulate massive matrices. Matrices are allocated to shared memory and may use memory-mapped files. Packages biganalytics, bigtabulate, synchronicity, and bigalgebra provide advanced functionality.

External Links:

CRAN repository

97 questions
27
votes
2 answers

Shared memory in parallel foreach in R

Problem Description: I have a big matrix c, loaded in RAM memory. My goal is through parallel processing to have read only access to it. However when I create the connections either I use doSNOW, doMPI, big.matrix, etc the amount to ram used…
Stanislav
  • 2,629
  • 1
  • 29
  • 38
15
votes
2 answers

Computing the null space of a bigmatrix in R

I can not find any function or package to calculate the null space or (QR decomposition) of a bigmatrix (from library(bigmemory)) in R. For example: library(bigmemory) a <- big.matrix(1000000, 1000, type='double', init=0) I tried the following but…
Mahin
  • 193
  • 8
8
votes
1 answer

Calculate Euclidean distance matrix using a big.matrix object

I have an object of class big.matrix in R with dimension 778844 x 2. The values are all integers (kilometres). My objective is to calculate the Euclidean distance matrix using the big.matrix and have as a result an object of class big.matrix. I…
Samuel-Rosa
  • 339
  • 3
  • 10
7
votes
2 answers

big.matrix as data.frame in R

I've recently started using R for data analysis. Now I've got a problem in ranking a big query dataset (~1 GB in ASCII mode, over my laptop's 4GB RAM in binary mode). Using bigmemory::big.matrix for this dataset is a nice solution, but providing…
Igor Shalyminov
  • 694
  • 2
  • 8
  • 22
5
votes
3 answers

Example of bigmemory and friends with file backing

I am interested in exploring how R can handle data out-of-memory. I've found the bigmemory package and friends (bigtabulate and biganalytics), but was hoping that someone could point me to a worked out example that uses file backing with these…
Charlie
  • 2,801
  • 3
  • 26
  • 27
5
votes
1 answer

Big matrix and memory problems

I am working on a huge dataset and I would like to derive the distribution of a test statistic. Hence I need to do calculations with huge matrices (200000x200000) and as you might predict I have memory issues. More precisely I get the following:…
Akis
  • 130
  • 1
  • 8
5
votes
2 answers

How to install BigMemory and bigrf on windows OS

I have been trying to install bigmemory on my R installation. My OS is windows 7 64 bit and I have tried it on R V2.15.1,2.15.2 and 3.0.1 64 bit but I cant get it to work. I have tried several options download the current source and run the…
Kharoof
  • 597
  • 1
  • 6
  • 21
5
votes
1 answer

R::bigmemory - how to create character big.matrix?

I try to use bigmemory package in R and I'm stuck in the very beginning. I do: temp <- matrix(paste("a",1:10), 5, 2) and get a character matrix. That's OK. But then I try: x <- as.big.matrix(temp, type="char") and I get a matrix full of NA and the…
user1890450
  • 53
  • 4
5
votes
1 answer

R: big.matrix multiplication

All, I am working with R2.14 on 64-bit Windows 7. I am trying to square a big.matrix object (big thanks to the developers), and may require raising it to a higher power. From my research the bigalgebra package would be right up my street, but it…
woody
  • 59
  • 2
5
votes
1 answer

R: Using the bigmemory library for classification with randomForest

has anyone been able to set up a classification (not a regressions) using randomForest AND the bigmemory library. I am aware that the 'formula approach" cannot be used and we have to resort to the "x=predictors, y=response approach". It appears that…
auro
  • 1,079
  • 1
  • 10
  • 22
4
votes
2 answers

Linear regression with big matrices

I would like to perform a linear regression with big matrices. This is what I have tried so far: library(bigmemory) library(biganalytics) library(bigalgebra) nrows <- 1000000 X <- as.big.matrix( replicate(100, rnorm(nrows)) ) y <-…
mat
  • 2,412
  • 5
  • 31
  • 69
4
votes
2 answers

R: Distm for big data? Calculating minimum distances between two matrices

I have two matrices, one is 200K rows long, the other is 20K. For each row (which is a point) in the first matrix, I am trying to find which row (also a point) in the second matrix is closest to the point in the first matrix. This is the first…
Ana
  • 421
  • 3
  • 12
4
votes
1 answer

R bigmemory always use backing file?

We are trying to use the BigMemory library with foreach to parallel our analysis. However, the as.big.matrix function seems always use backingfile. Our workstations have enough memory, is there a way to use bigMemory without the backing file? This…
user7648269
  • 111
  • 1
4
votes
1 answer

Memory problems using bigmemory to load large dataset in R

I have a large text file (>10 million rows, > 1 GB) that I wish to process one line at a time to avoid loading the entire thing into memory. After processing each line I wish to save some variables into a big.matrix object. Here is a simplified…
slabofguinness
  • 773
  • 1
  • 9
  • 19
4
votes
2 answers

Proper way to subset big.matrix

I would like to know if there is a 'proper' way to subset big.matrix objects in R. It is simple to subset a matrix but the class always reverts to 'matrix'. This isn't a problem when working with small datasets like this but with massive datasets…
cdeterman
  • 19,630
  • 7
  • 76
  • 100
1
2 3 4 5 6 7