0

I am trying to register below class for Kryo Serialization in spark code but I am receiving an error.

Code:

class KafkaSink(createProducer: () => KafkaProducer[String, String]) extends Serializable {
  lazy val producer = createProducer()
  def send(topic: String, key: String, value: String): Unit = producer.send(new ProducerRecord(topic, key, value))
}

object KafkaSink {
  def apply(config: Properties): KafkaSink = {
    val f = () => {
      val producer = new KafkaProducer[String, String](config)

      //close producer if VM exits
      sys.addShutdownHook {
        producer.close()
      }
      producer
    }
    new KafkaSink(f)
  }
}

Error:

Caused by: java.lang.IllegalArgumentException: Class is not registered: com.test.KafkaSink$$anonfun$1 Note: To register this class use: kryo.register(com.test.KafkaSink$$anonfun$1.class); at com.esotericsoftware.kryo.Kryo.getRegistration(Kryo.java:488) at com.esotericsoftware.kryo.util.DefaultClassResolver.writeClass(DefaultClassResolver.java:97) at com.esotericsoftware.kryo.Kryo.writeClass(Kryo.java:517) at com.esotericsoftware.kryo.serializers.ObjectField.write(ObjectField.java:76) ... 14 more

I have tried registering the class using below 2 approaches which didn't work and giving me same error

kryo.register(classOf[KafkaSink])
kryo.register(KafkaSink.getClass)

How can I register this class?

Deepak Janyavula
  • 348
  • 4
  • 17
  • This seems to have been answered here already: [Kryo serialization refuses to register class](https://stackoverflow.com/questions/22027451/kryo-serialization-refuses-to-register-class). Does this help you? – Tobi Oct 05 '18 at 15:48
  • 1
    I think the problem is that Kryo didn't know about your KafkaProducer creation function - this is the exeption is all about. So, try to do this: kryo.register(Class.forName("com.test.KafkaSink$$anonfun$1") kryo.register(classOf[KafkaSink]) – codejitsu Oct 05 '18 at 16:02
  • @leshkin is right as to the source of the problem: the error is definitely *not* about `KafkaSink` not being registered, but about the first anonymous function within `KafkaSink` (`$$annonfun$1`) not being registered. As to the solution, I don't know if @leshkin's solution will work or not, because I don't know if such a lambda is serializable or not. I suggest you make sure lambda serialization is really what you want - I'd be very reluctant to serialize a lambda (instead, I'd much prefer to serialize some "regular" object). – Tomasz Linkowski Oct 06 '18 at 13:07
  • @leshkin, Thanks. That really solved my issue. Your solution works. I struggled in solving this issue for months and now I have a working solution. Thanks a lot!!! – Deepak Janyavula Oct 08 '18 at 07:12

0 Answers0