Using Breeze from Java on Spark MLlib

Question

While trying to use MLlib from Java, what is the correct way to use breeze Matrix operations? For e.g. multiplication in scala it ist simply "matrix * vector". How is the corresponding functionality expressed in Java?

There are methods like "$colon$times" which might be invoked by the correct way

breeze.linalg.DenseMatrix<Double> matrix= ...
breeze.linalg.DenseVector<Double> vector = ...  

matrix.$colon$times( ...

one might need an operator instance ... breeze.linalg.operators.OpMulMatrix.Impl2 But which exact typed Operation instance and parameters are to be used?

You may try `common Math` for Java. – CᴴᴀZ Dec 28 '16 at 13:13 — CᴴᴀZ, Dec 28 '16 at 13:13

score 4 · Answer 1 · answered Dec 02 '14 at 19:36

4

It's honestly very hard. Breeze makes very very heavy use of implicits, and they just don't translate well into Java. We have some Java friendly wrappers for signal processing, but nothing for linear algebra. (I'd happily take a pull request that provided some support for wrapping things.)

answered Dec 02 '14 at 19:36

dlwh

2,257
11
23

ok, thanks for the offer, but how often will we get into such issues for the many potential matrix operations? Is this really feasible? Can this handle a matrix which consumes almost the entire RAM? Converting (and maybe persisting before) the MLlib matrix and loading it to a matrix library more usable from Java like commons-math or parallel colt will be the way to go while using Java on MLlib? – Mechanee Dec 19 '14 at 11:34

score 0 · Answer 2 · answered Apr 16 '19 at 13:32

0

Have a look at https://spark.apache.org/docs/2.2.0/api/java/org/apache/spark/mllib/linalg/BLAS.html, which contains many of the BLAS linear algebra operations. These can be used by both Scala and Java.

answered Apr 16 '19 at 13:32

GroovyP

798
1
6
15

Using Breeze from Java on Spark MLlib

2 Answers2

Linked