Allocate a big matrix

Question

I'm using the bigmemory package. I want to calculate w. My v length is 478000 and k length is 240500. The two matrix multiplication is w very large.

I run the code by loop, but it still is running and is not finished yet and I don't know if will give me the result or not.

I tried to calculate it without the for loop, but I got and error. Please any help to correct my code to make it fast.

v <-read.big.matrix('v.dat',type='double')
k <-read.big.matrix('k.dat',type='double')
m=length(v);
n=length(k);
for(i in 1:m)
{
    for(j in 1:n)
    {
       w[i,j] = 2 * cos(dt * v[i] * k[j]) - 2
    }
}

How I can define w before the loop because the size of w is very large I couldn't do like w <- matrix(nr,ncol).

score 2 · Answer 1 · answered Jan 18 '13 at 13:53

Preallocating a matrix can be done like this:

m = matrix(rep(0, number_or_rows*number_of_columns), 
     number_of_rows, number_of_columns))

This creates a matrix with the amount of rows and columns defined in the variables number_of_rows and number_of_columns, filled initially with all 0.

What is probably going to be a problem is that because w is equal in size to v and k, you might very well run into memory issues when filling w. You could solve this by also using a bigmemory matrix for w, or running your analysis in chunks.

w should be 478000 by 240500.How i can allocate 1TB I got error regarding the too many elements to allocate — user1977435, Jan 18 '13 at 17:33

Carl Witthoft · Answer 2 · 2013-01-18T18:12:11.877

1

agstudy is on the right track, but you could use outer here, as

w <- outer(v,k,FUN=function(x,y) 2*cos(x*y)-2 )

v<-runif(10)
k<-runif(10)
m=length(v);
n=length(k);
w<-matrix(nr=m,nc=n)
for(i in 1:m)
{
    for(j in 1:n)
    {
       w[i,j] = 2 * cos( v[i] * k[j]) - 2
    }
}

ww <- outer(v,k,function(x,y) 2*cos(x*y)-2)

Test: ww-w is a matrix of zeroes.

edited Jan 18 '13 at 18:12

answered Jan 18 '13 at 15:33

Carl Witthoft

20,573
9
43
73

@agstudy I just tried it and it worked exactly fine. See edit. – Carl Witthoft Jan 18 '13 at 18:10
+1 Yes it works.. for some reasons I tested your code with 2 matrix not vectors.. – agstudy Jan 18 '13 at 18:38
but my problem here w i can not define it as w<- matrix(nr=m, nc=n) since w is very lage matrix it works with me in small matrix but i need it to work with large matrix. My w here is 478000 by 240500 – user1977435 Jan 18 '13 at 19:51

score 1 · Answer 3 · answered Jan 18 '13 at 18:02

1

You need to use the 'big.matrix"-class constructors, and since you are obviously exceeding RAM resources, it would appear necessary that you define it as a "filebacked.big.matrix"

w <- filebacked.big.matrix( m, n , # additional arguments to allocate files and dims
                           )

See the last example in:

 help(big.matrix, package=bigmemory)

answered Jan 18 '13 at 18:02

IRTFM

258,963
21
364
487

I did it but the loop here didn't want to stop and the code was runnig from week ago till now and didn't get any results. – user1977435 Jan 18 '13 at 19:54
Did "it"? You mean you looked at the help page or you mean you executed similar code? You are not going to succeed with a regular matrix that is filling up virtual memory. Have you considered the possiblity this is just 'too big' for your computer? – IRTFM Jan 18 '13 at 20:18
I mean I tried to run this code for the big matrix but it didn't work with me I don't know what is the problem : library('bigmemory') library('irlba') v <-read.big.matrix('v.dat',type='double') k <-read.big.matrix('k.dat',type='double') m=length(v); n=length(k); w<-filebacked.big.matrix(m,n,type="double",backingfile="w.bin",descriptorfile="w.desc") for(i in 1:m) { for(j in 1:n) { w[i,j]=2*cos(0.001*v[i]*k[j])-2 } } – user1977435 Jan 19 '13 at 08:20

score 0 · Answer 4 · answered Jan 18 '13 at 14:55

0

I would do something like this using R vectorization feature:

for(i in 1:m)
{
  w[i] = 2 * cos(dt * v[i] * k) - 2 # I compute n terms here
}

answered Jan 18 '13 at 14:55

agstudy

119,832
17
199
261

I have to define w .I got this error message object 'w' not found. w here should be 478000 by 240500. my problem is in the memory – user1977435 Jan 18 '13 at 19:43

Allocate a big matrix

4 Answers4