0

I've read some similar posts, while none of them actually tackled my problem.

I need to do a series of multiplication-similar operations for A, B, specifically calculating kernel matrices, on Windows Platform. While, the problem is both of A, B could be really large, let us say, 20000-by-360000. While, my server can only provide 96 GB memory. It may seem infeasible to have them in memory at the same time and do the calculation. So is there any good way to efficiently handle such a large multiplication? Btw, The size of result, which is 20000-by-20000, is much less than the multiplier and can fit in the memory properly.

Because I do it on Windows, it may be not feasible to call functions like mmap2.

I wonder whether converting them into sparse matrix is a good option. However, it may heavily depend on the properties of data.

Another solution I've come up with is to partition the origin matrix into blocks. Then do the calculation block-by-block.

Is there any other better solution? Any practical suggestions would be really appreciated.

Best regards, Peiyun

Peiyun
  • 171
  • 1
  • 2
  • 13
  • 1
    possible duplicate of [Efficient multiplication of very large matrices in MATLAB](http://stackoverflow.com/questions/4420235/efficient-multiplication-of-very-large-matrices-in-matlab) – shoover Jan 14 '14 at 19:55
  • Blocking + Strassen's Algorithm is good for big dense matrices. For sparse ones, I dont know. – huseyin tugrul buyukisik Jan 15 '14 at 20:11
  • I think anyone interested in this problem may want to check the sparse matrix out. It saves memory and is compatible with mex-functions and built-in functions. Thanks @shoover – Peiyun Jan 17 '14 at 15:08

1 Answers1

0

If I where you I'd look into the block processing function:

B = blockproc(filename,[M N],fun)

and use the Destination parameter to allow saving the results without overflowing your memory.

Cape Code
  • 3,584
  • 3
  • 24
  • 45