Memory Requirement of Commons Math: The Apache Commons Mathematics Library

Question

I would like to know if all the operations in Apache Commons Math are in memory operations. I am particularly interested in OLSMultipleLinearRegression and using this for big data.

Also is there any existing JAVA API for running regression on big data.

Stephen's answer below looks pretty conclusive, but I'd suggest you also email the [commons user mailing list](http://commons.apache.org/mail-lists.html), prefixing your subject with `[math`], if you'd like to verify that with the developers. You'll get a quick response I'm sure (which you could post here for posterity). — Duncan Jones, Mar 27 '14 at 09:56

Stephen C · Answer 1 · 2014-03-27T10:01:45.973

1

My reading of the javadocs for the OLSMultipleLinearRegression class is that it is entirely in memory. For a start, the model and the sample data must be supplied as primitive arrays which precludes any possibility of representations that don't fit in memory ... at least at the point where you provide the inputs.

edited Mar 27 '14 at 10:01

answered Mar 27 '14 at 09:52

Stephen C

698,415
94
811
1,216

Thank you Stephen and Duncan for your inputs. Is there any API which allows regression on big data without using memory. – unnik Mar 27 '14 at 10:16
I couldn't find one, but I didn't look very hard. – Stephen C Mar 27 '14 at 11:14

Phil Steitz · Answer 2 · 2015-03-21T22:26:41.140

0

It is correct that OLSMultipleLinearRegression stores the full design matrix in memory. There is another OLS implementation class, however, MillerUpdatingRegression, in the same Commons Math package (version 3.0 onward) that does not store the input dataset in memory.

edited Mar 21 '15 at 22:26

answered Mar 21 '15 at 22:19

Phil Steitz

644
3
10

Memory Requirement of Commons Math: The Apache Commons Mathematics Library

2 Answers2