0

I am trying to multiply a dense matrix A for its transpose A'. The matrix is about 2 million rows and 4 hundred columns. I implemented the multiplication in hadoop map reduce, but it runs too slowly because of the non locality of the job (every record must be multiplied by every other record). Therefore, I am trying to use Apache Hama for the multiplication. There is a class called DenseDoubleMatrix, but I think it is not useful because it seems that it does not distribute the calculations across the cluster using bsp.

Is there any other facility already implemented in Hama for matrix multiplication or do I have to implement it myself using the bsp model?

giulatona
  • 137
  • 2
  • 9
  • Analyze if you really need that product or if an algorithm using the QR decomposition of A or the SVD for greater numerical stability is available. The product A*A' is numerically bad since it squares the condition number of A. – Lutz Lehmann Feb 26 '14 at 18:49
  • I need that product, because I need to compute the dot product between all the rows of A – giulatona Feb 27 '14 at 15:43
  • So you compute the Gram matrix. And then, what do you do with it? For example, for least square problems it is better to use the QR decomposition of A. In general, it is better to first compute the QR decomposition, here of A'. Then A*A'=R'*Q'*Q*R=R'*R and R is a smaller and square matrix. – Lutz Lehmann Feb 27 '14 at 15:47
  • For distributed computing using the plain matrix multiplication you could try to divide the matrix into 400x400 blocks. Then you still have 5000x5000 matrix block multiplications to distribute and each task has a useful size so that communication overhead is small relative to the size of data and computation time. If you are already doing that, please indicate so in the question. – Lutz Lehmann Feb 27 '14 at 15:51
  • @LutzL,A = QR, A*A' = QR * (QR)' = QRR'Q' != R'R – liuyang1 Mar 01 '14 at 10:03
  • @Procras: That is why I wrote that you have to decompose the transpose A'=QR. – Lutz Lehmann Mar 01 '14 at 10:12
  • @LutzL,you are RIGHT! Sorry, I did not see clearly. but I test it, it seems not faster than directly multiply. Test it using numpy. – liuyang1 Mar 01 '14 at 10:35

0 Answers0