2

I have a simple Spark application with Scala and SBT. First I tried to do the following:

  • run sbt clean package

  • run spark-submit --class Main ./target/scala-2.11/sparktest_2.11-1.0.jar

but it fails with the following exception:

Exception in thread "main" java.lang.NoClassDefFoundError: com/fasterxml/jackson/module/scala/DefaultScalaModule$

Then I tried the assembly plugin for SBT, but I got the following exception instead:

java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.introspect.POJOPropertyBuilder.addField(Lcom/fasterxml/jackson/databind/introspect/AnnotatedField;Lcom/fasterxml/jackson/databind/PropertyName;ZZZ)V

As I can see, everything looks related to the Jackson lib and to the Scala support. Maybe it's some issue related to versions of the libraries?

My build.sbt looks like this:

name := "SparkTest"

version := "1.0"

scalaVersion := "2.11.4"

scalacOptions := Seq("-unchecked", "-deprecation", "-encoding", "utf8", "-feature")

libraryDependencies ++= {
  Seq(
    "org.apache.spark"              %% "spark-core"           % "1.2.1"     % "provided",
    "com.fasterxml.jackson.core"    % "jackson-core"          % "2.4.1",
    "com.fasterxml.jackson.core"    % "jackson-databind"      % "2.4.1",
    "com.fasterxml.jackson.module"  %% "jackson-module-scala" % "2.4.1"
  )
}

And my application code is simply this:

import com.fasterxml.jackson.databind.{DeserializationFeature, ObjectMapper}
import com.fasterxml.jackson.module.scala.DefaultScalaModule
import org.apache.spark.{SparkConf, SparkContext}

trait JsonUtil {
  val mapper = new ObjectMapper()
  mapper.registerModule(DefaultScalaModule)
  mapper.configure(DeserializationFeature.FAIL_ON_UNKNOWN_PROPERTIES, false)
}

case class Person(name: String)

object Main extends JsonUtil {

  def main(args: Array[String]) {

    val conf = new SparkConf().setAppName("Spark Test App")
    val sc = new SparkContext(conf)
    val inputFile = "/home/user/data/person.json"
    val input = sc.textFile(inputFile)

    val persons = input.flatMap { line ⇒ {
      try {
        println(s"  [DEBUG] trying to parse '$line'")
        Some(mapper.readValue(line, classOf[Person]))
      } catch {
        case e : Exception ⇒
          println(s"  [EXCEPTION] ${e.getMessage}")
          None
      }
    }}

    println("PERSON LIST:")
    for (p ← persons) {
      println(s"  $p")
    }
    println("END")
  }

}

EDIT: the problem seems to be related to the Spark application. If I run simple application just for testing JSON unmarshalling everything goes OK. But if I try to do the same from the Spark application, then the problem appears as described above. Any ideas?

ale64bit
  • 6,232
  • 3
  • 24
  • 44
  • Try putting your code in a package. – Soumya Simanta Mar 09 '15 at 01:59
  • @SoumyaSimanta Like throwing `JsonUtil`, `Person` and `Main` in some common package? – ale64bit Mar 09 '15 at 02:01
  • @SoumyaSimanta Just tried and the problem persist. Same exceptions. – ale64bit Mar 09 '15 at 02:06
  • I can run your codes without exception on latest Spark trunk. What Spark version you use? 1.2.1? – viirya Mar 09 '15 at 08:21
  • "scalaVersion := "2.11.4". From the Spark 1.2.1 page: "For the Scala API, Spark 1.2.1 uses Scala 2.10. You will need to use a compatible Scala version (2.10.x)." – The Archetypal Paul Mar 09 '15 at 09:49
  • @Paul The Spark distribution I have, is the one built from sources (1.2.1) following instructions for building for Scala 2.11. The mistake can be originated because of this anyway? – ale64bit Mar 09 '15 at 11:28
  • 1
    i assumed you were using the distributed files, not building from source, since your question didn't say otherwise. I think it might be an idea to try it with the prebuilt files (and scala 2.10.x) - then you'll know if it's a library/version problem. The other questions (where the issue was a version conflict) do seem to describe something similar to your situation – The Archetypal Paul Mar 09 '15 at 16:14
  • 1
    @Paul Downgraded to scala 2.10.4 and prebuilt spark 1.2.1 and the problem disappeared. Thanks a lot)) – ale64bit Mar 09 '15 at 22:16

0 Answers0