sbt assembly plugin includes "provided" libraries and it's lineage

Question

Here is my build.sbt

resolvers  ++= Seq("Apache Development Snapshot Repository" at "https://repository.apache.org/content/repositories/snapshots/",
  Resolver.mavenLocal)

name := "flink-wiki-edits"

version := "1.0"

scalaVersion := "2.11.7"

val flinkVersion = "1.1.3"

val flinkDependencies = Seq(
  "org.apache.flink" %% "flink-scala" % flinkVersion % "provided",
  "org.apache.flink" %% "flink-streaming-scala" % flinkVersion % "provided",
  "org.apache.flink" %% "flink-connector-wikiedits"% flinkVersion )

lazy val root = (project in file(".")).
  settings(
    libraryDependencies ++= flinkDependencies
  )

mainClass in assembly := Some("com.ta.WikipediaAnalytics")

// make run command include the provided dependencies
run in Compile <<= Defaults.runTask(fullClasspath in Compile, mainClass in (Compile, run), runner in (Compile, run))

// exclude Scala library from assembly
assemblyOption in assembly := (assemblyOption in assembly).value.copy(includeScala = false)

and the console output snippet of $ sbt assembly

vagrant@vagrant-ubuntu-trusty-64:/workspace/flink-wiki-edits$ sbt assembly
[info] Loading project definition from /workspace/flink-wiki-edits/project
[info] Set current project to flink-wiki-edits (in build file:/workspace/flink-wiki-edits/)
[info] Including: flink-connector-wikiedits_2.11-1.1.3.jar
[info] Including: flink-streaming-java_2.11-1.1.3.jar
[info] Including: flink-core-1.1.3.jar
[info] Including: flink-annotations-1.1.3.jar
[info] Including: jsr305-1.3.9.jar
[info] Including: commons-lang3-3.3.2.jar
[info] Including: slf4j-api-1.7.7.jar
[info] Including: slf4j-log4j12-1.7.7.jar
[info] Including: log4j-1.2.17.jar
[info] Including: force-shading-1.1.3.jar
[info] Including: flink-metrics-core-1.1.3.jar
[info] Including: kryo-2.24.0.jar
[info] Including: minlog-1.2.jar

where I see lot of libraries that are part of the

"org.apache.flink" %% "flink-scala" % flinkVersion % "provided",
"org.apache.flink" %% "flink-streaming-scala" % flinkVersion % "provided"

lineage, I assumed that provided will exclude the libraries while assembling the fat-jar, let me know if I have overlooked something.

Do you see them in the output JAR? – Yuval Itzchakov Jan 19 '17 at 10:33 — Yuval Itzchakov, Jan 19 '17 at 10:33
Yes, I do see them – Somasundaram Sekar Jan 19 '17 at 10:43 — Somasundaram Sekar, Jan 19 '17 at 10:43

score 1 · Accepted Answer · answered Jan 19 '17 at 11:14

This is caused by flink-connector-wikiedits library has dependency flink-streaming-java_2.11, see: flink-streaming-java_2.11 pom:

   <dependency>
      <groupId>org.apache.flink</groupId>
      <artifactId>flink-streaming-java_2.11</artifactId>
      <version>1.1.3</version>
      <scope>compile</scope>
    </dependency>

and also flink-streaming-java_2.11 has flink-core ... dependency, see:

http://search.maven.org/#artifactdetails%7Corg.apache.flink%7Cflink-streaming-java_2.11%7C1.1.3%7Cjar

sbt assembly plugin includes "provided" libraries and it's lineage

1 Answers1