0

I want to register kryo Serialization, for that i have written below code,

First Class (Select.Class)

var SparkConf = new SparkConf()
      .set("spark.kryo.registrationRequired", "true")
      .set("spark.kryo.registrator", "org.agg.MyRegistrator")

var sc = new SparkContext(SparkConf)
var sqlContext = new org.apache.spark.sql.hive.HiveContext(sc)


val testdata_select = sqlContext.sql("select a,b, from test")

Second Class (MyRegistrator.Class)

class MyRegistrator extends KryoRegistrator {
  override def registerClasses(kryo: Kryo) {
    kryo.register(classOf[TransformationCode])
  } 
}

Third Class (TransformationCode.Class)

//Transformation Code

But it gives me error that

User class threw exception: org.apache.spark.SparkException: Task not serializable

MichaelChirico
  • 33,841
  • 14
  • 113
  • 198
Darshan Manek
  • 155
  • 1
  • 2
  • 12
  • How are you using `Select.Class` in your code? Show us the definition of your Spark graph. – Yuval Itzchakov Oct 10 '16 at 12:28
  • you shouldn't register your transformation code, you should register you data classes. The classes that hold the data that need to be moved around during shuffle, collect, etc. – marios Oct 11 '16 at 02:58

0 Answers0