How to troubleshoot SBT's library dependency warnings?

Question

I'm trying to build a "hello world"-esque app that uses Spark streaming to stream data from a Kafka broker (this works), filters/processes this data, and pushes it to a (local) web browser using the Scalatra web framework and its supported web sockets functionality from Atmosphere. The Kafka/Spark chunk works independently, and the Scalatra/Atmosphere chunk also works independently. It's when I try to bring the two halves together that I run into issues with library dependencies.

The real question: how do I go about selecting library versions for which Spark will play nice with Scalatra?

A bare bones Scalatra/Atmosphere app works fine as follows:

organization := "com.example"
name := "example app"
version := "0.1.0"
scalaVersion := "2.12.2"

val ScalatraVersion = "2.5.+"

libraryDependencies ++= Seq(
  "org.json4s"                  %% "json4s-jackson"      % "3.5.2",
  "org.scalatra"                %% "scalatra"            % ScalatraVersion,
  "org.scalatra"                %% "scalatra-scalate"    % ScalatraVersion,
  "org.scalatra"                %% "scalatra-specs2"     % ScalatraVersion    % "test",
  "org.scalatra"                %% "scalatra-atmosphere" % ScalatraVersion,
  "org.eclipse.jetty"           %  "jetty-webapp"        % "9.4.6.v20170531"  % "provided",
  "javax.servlet"               %  "javax.servlet-api"   % "3.1.0"            % "provided"
)

enablePlugins(JettyPlugin)

But if I add new dependencies for Spark and Spark streaming, and knock the Scala version down to 2.11 (required for Spark-Kafka streaming):

organization := "com.example"
name := "example app"
version := "0.1.0"
scalaVersion := "2.11.8"

val ScalatraVersion = "2.5.+"
val SparkVersion = "2.2.0"

libraryDependencies ++= Seq(
  "org.json4s"                  %% "json4s-jackson"      % "3.5.2",
  "org.scalatra"                %% "scalatra"            % ScalatraVersion,
  "org.scalatra"                %% "scalatra-scalate"    % ScalatraVersion,
  "org.scalatra"                %% "scalatra-specs2"     % ScalatraVersion    % "test",
  "org.scalatra"                %% "scalatra-atmosphere" % ScalatraVersion,
  "org.eclipse.jetty"           %  "jetty-webapp"        % "9.4.6.v20170531"  % "provided",
  "javax.servlet"               %  "javax.servlet-api"   % "3.1.0"            % "provided"
)

libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % SparkVersion,
  "org.apache.spark" %% "spark-streaming" % SparkVersion,
  "org.apache.spark" %% "spark-streaming-kafka-0-8" % SparkVersion
)

enablePlugins(JettyPlugin)

The code compiles, but I get SBT's eviction warning:

[warn] There may be incompatibilities among your library dependencies.
[warn] Here are some of the libraries that were evicted:
[warn]  * org.json4s:json4s-jackson_2.11:3.2.11 -> 3.5.3
[warn] Run 'evicted' to see detailed eviction warnings

Then finally, when Jetty tries to run the web server, it fails with this error:

WARN:oejuc.AbstractLifeCycle:main: FAILED org.eclipse.jetty.annotations.ServletContainerInitializersStarter@53fb3dab: java.lang.NoClassDefFoundError: com/sun/jersey/spi/inject/InjectableProvider
java.lang.NoClassDefFoundError: com/sun/jersey/spi/inject/InjectableProvider

How do I get to the bottom of this? I'm new to the Scala world, and the intricacies of dependencies are blowing my mind.

Using the dependency-graph plugin (https://github.com/jrudolph/sbt-dependency-graph) would probably help to find why the eviction warning is coming up (spark probably depends some lower/higher version of jackson than is being improted elsewhere). You can use the exclude keyword to get rid of the offending import: (see http://www.scala-sbt.org/0.13/docs/Library-Management.html#Exclude+Transitive+Dependencies). If this didn't resolve it, it might be worth pushing a minimum version of whatever you have to GitHub, so we can dive into the issue more easily. — prince, Jul 30 '17 at 08:28
Thanks @prince. I think the issue may be related to jetty and/or javax servlet after switching to scala 2.11. I pushed a barebones example to github, with an explanation of the flow of changes in build.sbt comments. I'd be really curious if you can understand the issue. After compiling, the 'evicted' task actually produces a long list of issues (not just json4s-jackson as I had thought). https://github.com/ian-wright/scalatra-atmosphere-spark-example/blob/master/build.sbt — EyeWrite, Aug 01 '17 at 15:52

score 0 · Answer 1 · answered Aug 01 '17 at 06:49

0

One way to remove the eviction warning is to add the library dependency with the required version using dependencyOverrides

try to add the following in your SBT file and re-build the application

dependencyOverrides += "org.json4s" % "json4s-jackson_2.11" % "3.5.3"

Check SBT documentation here

answered Aug 01 '17 at 06:49

Prasad Khode

6,602
11
44
59

Thanks Prasad. I'm thinking that the issue is related to jetty and/or the javax servlet that Scalatra uses, after I switched to Scala 2.11 (not just a single dependency like json4s-jackson_2.11). I pushed a simple example of the project to github, with an explanation of the offending changes in build.sbt, if you're up to taking a peak at it: https://github.com/ian-wright/scalatra-atmosphere-spark-example/blob/master/build.sbt – EyeWrite Aug 01 '17 at 16:13

How to troubleshoot SBT's library dependency warnings?

1 Answers1