0

I'm using weka.mi.MISVM in a Scala/Spark program and need to serialize my kernels to reuse them later.

For this I use Kryo as this :

conf.set("spark.serializer", "org.apache.spark.serializer.KryoSerializer")
conf.registerKryoClasses(
    Array(classOf[Multiset], classOf[MISVM], classOf[(_,_)],
     classOf[Map[_,_]], classOf[Array[_]])
)

...

val patterns: RDD[(Multiset, MISVM)] = ...
patterns.saveAsObjectFile(options.get.out)

(Multiset is one of my objects)

The serialization works well but when I tried to read my kernels with:

objectFile[(Multiset, MISVM)](sc, path)

I obtain this error: 

com.esotericsoftware.kryo.KryoException: Encountered unregistered class ID: 13994

I think that it's because I don't have register all the classes used by MISVM and I read that Kryo.setRegistrationRequired(false) could be a solution but I don't understand how to use it in my case.

How to define that the conf KryoSerializer have to use setRegistrationRequired(false)?

Heleo
  • 25
  • 8

1 Answers1

0

Try this:

conf.set("spark.serializer", org.apache.spark.serializer.KryoSerializer")
conf.set("spark.kryo.registrationRequired", "false")
Dima Ogurtsov
  • 1,487
  • 1
  • 10
  • 18
  • Thanks I get an error notebook:1: error: not found: value conf . What needs to be imported? – Sade Oct 19 '18 at 10:00
  • @Sade conf here is spark's configuration, get it whatever way you get spark configuration. Something like spark().conf() or spark().configuration() or something like it – Dima Ogurtsov Oct 21 '18 at 05:58