0

I am working on https://github.com/capitalone/Hydrograph. While upgrading Spark version to 2.2.1, I am seeing below error at line 89 in

https://github.com/capitalone/Hydrograph/blob/master/hydrograph.engine/hydrograph.engine.spark/src/main/scala/hydrograph/engine/spark/flow/HydrographRuntime.scala

Caused by: java.lang.NoClassDefFoundError: com/esotericsoftware/kryo/io/UnsafeOutput
  at org.apache.spark.serializer.SerializerManager.<init>(SerializerManager.scala:42)
  at org.apache.spark.SparkEnv$.create(SparkEnv.scala:289)
  at org.apache.spark.SparkEnv$.createDriverEnv(SparkEnv.scala:175)
  at org.apache.spark.SparkContext.createSparkEnv(SparkContext.scala:257)
  at org.apache.spark.SparkContext.<init>(SparkContext.scala:432)
  at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2516)
  at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:918)
  at org.apache.spark.sql.SparkSession$Builder$$anonfun$6.apply(SparkSession.scala:910)
  at scala.Option.getOrElse(Option.scala:121)
  at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:910)

I don't have much support for this tool now and recently took over working on it. Any pointers for solution of this problem will be helpful?

SOLUTION: Add following dependency in build.gradle file

compile group: 'com.esotericsoftware', name: 'kryo', version: '4.0.1'

MichaelChirico
  • 33,841
  • 14
  • 113
  • 198
ashutosh mittal
  • 46
  • 6
  • Those links are totally dodgy, and it is likely the answer was removed for a reason, and doesn't answer your question anyway. You should just re-ask the question, providing the necessary details as per [ask]. –  Feb 26 '18 at 21:00
  • Thanks, i will update the question with specific details – ashutosh mittal Feb 26 '18 at 21:36

1 Answers1

0

The error seems to be pretty straightforward, it's not able to find kryo libraries which might have been used in spark config and as such spark is unable to create a session.

Kumar Vaibhav
  • 2,632
  • 8
  • 32
  • 54