Large Matrix Multiplication using PDL - Perl

Question

I need to generate (40000 x 20000) matrix via SQL and perform matrix operation like A'A. Then I need to obtain Eigen value and Eigen Vectors.

Can you suggest me ways to implement this in Perl. (Maybe via PDL)

How can I multiply large matrix stored in file.?

Thank You

This looks a lot to me like you're trying to implement something like least squares fitting using a naive algorithm. PDL can do the operation you want, but perhaps you could describe the actual problem you're trying to solve. PDL may have a canned solution to that. — David Mertens, Sep 06 '14 at 11:30

score 1 · Answer 1 · answered Mar 10 '22 at 20:39

Using the latest PDL, you can assemble various pieces to achieve what you're aiming for:

https://metacpan.org/pod/PDL::IO::FastRaw#mapfraw to memory-map a disk file as a PDL ndarray, with no RAM limitations
https://metacpan.org/pod/PDL::IO::DBI to read from a DBI handle
https://metacpan.org/pod/PDL::LinearAlgebra to use LAPACK routines for various fast matrix operations including eigenvalues - make sure you either turn off PDL auto-parallelism if you use a parallel LAPACK such as OpenBLAS

If David is right and you are actually aiming to do least-squares fitting, PDL::LinearAlgebra has routines for that as well.

Mark Baker · Answer 2 · 2016-01-31T06:33:07.183

0

$PDL::BIGPDL=1; $c = sequence(60000, 60000); ## you need 32gb of ram and about 75gb of swap to do this.

$c * $d is different then $c x $d in PDL I believe there are modules for Eigen value or Eigen vector I have not used them yet tho...

edited Jan 31 '16 at 06:33

answered Dec 08 '14 at 16:34

Mark Baker

If you're the same Mark Baker who created [this electron simulation](https://perlmonks.org/?node_id=963819), it's just been incorporated in git `master`'s `demo 3dgal`, to be released soon! – Ed. Mar 10 '22 at 20:41

2 Answers2