How to serialize apache spark's MatrixFactorizationModel in Java

Question

I am building a recommendation system using Apache Spark MLlib and Java. Once the MatrixFactorizationModel is built, I have serialized it as a java object and when retrieving the model, I am getting the following exception.

Caused by: java.lang.ClassNotFoundException: org.apache.spark.OneToOneDependency cannot be found by org.scala-lang.scala-library_2.10.4.v20140209-180020-VFINAL-b66a39653b

Any idea how to fix this?

which version of spark are you using? – Jishnu Prathap Aug 17 '15 at 11:02 — Jishnu Prathap, Aug 17 '15 at 11:02
what version of scala is in your dependency? – GameOfThrows Aug 17 '15 at 14:24 — GameOfThrows, Aug 17 '15 at 14:24
scala 2.10.4 is used in the dependency – madawa Aug 21 '15 at 08:12 — madawa, Aug 21 '15 at 08:12

score 1 · Accepted Answer · answered Aug 18 '15 at 06:57

1

You are usiong incompatible versions of scala and spark. Spark 1.4.0 need scala version 2.11.6 but are using scala version 2.10.4(from your error stack).

PS:You could find out compatible versions from here

answered Aug 18 '15 at 06:57

Jishnu Prathap

1,955
2
21
37

How to serialize apache spark's MatrixFactorizationModel in Java

1 Answers1