0

I have written the following sbt file

name := "Test"

version := "1.0"

scalaVersion := "2.11.7"

libraryDependencies ++= Seq(
  "org.apache.hadoop" % "hadoop-client" % "2.7.1",
  "org.apache.spark" % "spark-core_2.10" % "1.3.0",
  "org.apache.avro" % "avro" % "1.7.7",
  "org.apache.avro" % "avro-mapred" % "1.7.7"
)

mainClass := Some("com.test.Foo")

I also have the following assembly.sbt file in projects folder

resolvers += Resolver.url("bintray-sbt-plugins", url("http://dl.bintray.com/sbt/sbt-plugin-releases"))(Resolver.ivyStylePatterns)
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.0")

when i do sbt assembly i get a huge list of errors

[error] (*:assembly) deduplicate: different file contents found in the following:
[error] /Users/abhishek.srivastava/.ivy2/cache/com.esotericsoftware.kryo/kryo/bundles/kryo-2.21.jar:com/esotericsoftware/minlog/Log$Logger.class
[error] /Users/abhishek.srivastava/.ivy2/cache/com.esotericsoftware.minlog/minlog/jars/minlog-1.2.jar:com/esotericsoftware/minlog/Log$Logger.class
[error] deduplicate: different file contents found in the following:
[error] /Users/abhishek.srivastava/.ivy2/cache/com.esotericsoftware.kryo/kryo/bundles/kryo-2.21.jar:com/esotericsoftware/minlog/Log.class
[error] /Users/abhishek.srivastava/.ivy2/cache/com.esotericsoftware.minlog/minlog/jars/minlog-1.2.jar:com/esotericsoftware/minlog/Log.class
[error] deduplicate: different file contents found in the following:
Knows Not Much
  • 30,395
  • 60
  • 197
  • 373

2 Answers2

1

I was able to resolve the problem. actually there is no need to build a fat jar because the "spark-submit" tool will have everything in the classpath anyway.

thus the right way to build the jar file is

name := "Test"

version := "1.0"

scalaVersion := "2.11.7"

libraryDependencies ++= Seq(
  "org.apache.hadoop" % "hadoop-client" % "2.7.1" % "provided",
  "org.apache.spark" % "spark-core_2.10" % "1.3.0" % "provided",
  "org.apache.avro" % "avro" % "1.7.7" % "provided",
  "org.apache.avro" % "avro-mapred" % "1.7.7" % "provided"
)

mainClass := Some("com.test.Foo")
Knows Not Much
  • 30,395
  • 60
  • 197
  • 373
1

1. use the MergeStrategy, see sbt-assembly

2. exclude the duplicated jars, such as:

    lazy val hbaseLibSeq = Seq(
    ("org.apache.hbase" % "hbase" % hbaseVersion).
      excludeAll(
        ExclusionRule(organization = "org.slf4j"),
        ExclusionRule(organization = "org.mortbay.jetty"),
        ExclusionRule(organization = "javax.servlet")),
    ("net.java.dev.jets3t" % "jets3t" % "0.6.1" % "provided").
      excludeAll(ExclusionRule(organization = "javax.servlet"))
  )

3. use the provided scope

show dependency tree:

  • ➜ cat ~/.sbt/0.13/plugins/plugins.sbt

addSbtPlugin("net.virtual-void" % "sbt-dependency-graph" % "0.7.5") addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.11.2")

  • ➜ cat ~/.sbt/0.13/global.sbt

net.virtualvoid.sbt.graph.Plugin.graphSettings

  • ➜ sbt dependency-graph
ljp
  • 135
  • 5