10

I am trying to calculate inverse of a very large matrix (11300x21500) in C++. So far I have tried Eigen and Armadillo libraries but both failed at initialization stage, saying that there is not enough memory. Can there be any way to overcome this situation?

Thanks in advance

P.S
I should correct the size of the matrix to 21500x21500. As UmNyobe suggested, this is not a square matrix. It is actually the observation matrix, X, and I am trying to calculate (XTX)-1

I have a 8GB memory(in a 64bit system), but I don't think I am making use of all of this memory space. The task manager shows that the memory usage at the time of error is 1GB. Maybe there is a OS command in Windows7 that closes an application when its memory usage exceeds 1GB.

By the way, my original purpose is to run a regression over this observation matrix.

One more thing: most columns in each row of the observation matrix X are zero. Can there be a way to take advantage of this, to limit the memory usage in the inverting operation?

Mat
  • 202,337
  • 40
  • 393
  • 406
Osman Darin
  • 101
  • 1
  • 2
  • 6
  • 3
    why are your dimensions not equals?? – UmNyobe May 09 '12 at 22:30
  • 2
    That matrix holds roughly 1GB or 2GB of data depending on whether you have 4- or 8-byte matrix entries. Are you on a 32-bit machine? – Steve Townsend May 09 '12 at 22:34
  • Steve I was going to post about the memory, you should write it in more detail as you mentioned it first. – UmNyobe May 09 '12 at 22:37
  • Do you mean a pseudoinverse? And why do you wish to calculate an inverse? If it's for, say, linear regression, I would suggest other techniques. – Robert Cooper May 09 '12 at 22:49
  • I think you need to tell us more about the machine you're trying to perform this calculation on - e.g. memory, 32 bit or 64 bit etc. Flag to re-open when you've done that. – Kev May 10 '12 at 00:59
  • @Robert Cooper Yes, my purpose is to run a regression. Is there an alternative technique? That would really be most helpful – Osman Darin May 10 '12 at 08:17
  • @RobertCooper Yes, my purpose is to run a regression. Is there an alternative technique? That would really be most helpful – Osman Darin May 11 '12 at 15:02
  • I see from stackoverflow.com/questions/2197623 that you've been here before :-). For very high dimensions, hill climbing may be faster. – Robert Cooper May 11 '12 at 17:58
  • @RobertCooper that's definitely not me:) but thanks, it's been a helpful link. do you recommend LAPACK btw? – Osman Darin May 11 '12 at 20:30
  • Actually, I've never used LAPACK. I'll defer to the more knowledgeable responders to [the C++ linear regression question](http://stackoverflow.com/questions/2197623) – Robert Cooper May 11 '12 at 21:03
  • You may want to know about the [Scientific Computation beta site](http://scicomp.stackexchange.com/), where you will presumably find a higher concentration of experts in these matters. – dmckee --- ex-moderator kitten May 12 '12 at 04:13

5 Answers5

6

Supposing the matrix is square, what you're probably looking for is an in-place matrix inversion algorithm.

You should check out this.

Community
  • 1
  • 1
Eitan T
  • 32,660
  • 14
  • 72
  • 109
5

You cannot inverse a non-square matrix.

http://en.wikipedia.org/wiki/Invertible_matrix

MrWuf
  • 1,478
  • 1
  • 14
  • 30
5

Assuming a (11300 x 11300) Matrix of integer (32 bits), you have

4*(11300^2)/(1024^3) = 0.4757 GB

If you are using double precision then double this number.

If the library is using the Strassen algorithm, which requires additional memory of the same magnitude, then you double the previous number.

So inverting a double-based matrix of this size with Strassen or gaussian will cost you 1.9 GB.

UmNyobe
  • 22,539
  • 9
  • 61
  • 90
  • So he need to provide details of the machine he is working on. He will not be able to inverse a 21500x21500 on a 32 bit machine for instance... – UmNyobe May 09 '12 at 23:12
  • 1
    Thanks for the input. As my machine specification, I have 8gb memory(in a 64bit system). But as far as I can keep track of the memory usage with task manager, windows closes the application when the used memory is 1gb. At least, that's the amount that the task managers shows. And that actually happens before taking the inverse. The application gives the error at the initialization stage of the matrix – Osman Darin May 10 '12 at 11:41
2

I'd like to propose another solution, which only works if you are not interested in the inverse of the matrix itself but in the product of the inverse with a vector. For example, assume that you would like to find the product of your inverse times a vector v, i.e. w := (X^T X)^{-1} v. In this case, you are actually looking for a solution to the problem

Find w such that (X^T X) w = v

Using iterative algorithms, it is possible to find w given X and v in the equation above without inverting X. One possibility that comes to my mind is using the Method of Conjugate Gradients. This algorithm can be implemented in roughly 10 lines and only requires to be able to compute the product (X^T X) y with a given vector y. In our case, this can even be done in two steps, i.e. compute z := X y and in a second step X^T z, which will save space as you do not need to store the product X^T X.

Mehrwolf
  • 8,208
  • 2
  • 26
  • 38
0

Although you're compiling your program on a 64-bit machine, you should also make sure that you are using the correct 64-bit libraries. Otherwise, the program may be compiled in 32-bit and you'll still get the same memory problems.

As for the computation of the inverse, the inverse function of OpenCV may help. Make sure to use the DECOMP_SVD inverse, as I found it to be more effective with near singular matrices.

Omid Sakhi
  • 140
  • 1
  • 5