sbt shading two versions of a dependency

Question

When I run my tests I see the following error:

Exception: java.lang.NoSuchMethodError: fs2.Stream$.bracket(Ljava/lang/Object;Lscala/Function1;Lscala/Function1;)Lfs2/internal/FreeC;

I first made sure my build file was cleaned up by explicitly declaring transitive dependencies and by removing unused dependencies to produce the following build.sbt:

scalaVersion := "2.12.7"
organization := "com.example "

lazy val lambdas = (project in file("."))
  .settings(
    libraryDependencies += "org.typelevel" %% "cats-core"   % "1.4.0",
    libraryDependencies += "org.typelevel" %% "cats-effect" % "1.0.0",

    libraryDependencies += "io.circe" %% "circe-core"    % "0.10.0",
    libraryDependencies += "io.circe" %% "circe-generic" % "0.10.0",
    libraryDependencies += "io.circe" %% "circe-parser"  % "0.10.0", 
    libraryDependencies += "io.circe" %% "circe-fs2"     % "0.10.0",

    libraryDependencies += "co.fs2" %% "fs2-core" % "1.0.0",
    libraryDependencies += "co.fs2" %% "fs2-io"   % "0.10.0",

    libraryDependencies += "org.http4s" %% "http4s-circe"  % "0.18.0",
    libraryDependencies += "org.http4s" %% "http4s-client" % "0.18.0",
    libraryDependencies += "org.http4s" %% "http4s-core"   % "0.18.0",

    libraryDependencies += "com.chuusai" %% "shapeless" % "2.3.3",

    libraryDependencies += "com.amazonaws" % "aws-lambda-java-core" % "1.2.0",
    libraryDependencies += "com.amazonaws" % "aws-java-sdk-core"    % "1.11.422",
    libraryDependencies += "com.amazonaws" % "aws-java-sdk-s3"      % "1.11.422",

    libraryDependencies += "org.scalatest"  %% "scalatest"  % "3.0.5"  % "test",
    libraryDependencies += "org.scalacheck" %% "scalacheck" % "1.14.0" % "test",

    libraryDependencies += "io.findify" %% "s3mock" % "0.2.4" % "test"
  )

Everything compiles and one of my two tests pass. The other fails with the error above. Here's the output of sbt evicted:

[warn] Found version conflict(s) in library dependencies; some are suspected to be binary incompatible:
[warn]  * co.fs2:fs2-core_2.12:1.0.0 is selected over 0.10.0
[warn]      +- io.circe:circe-fs2_2.12:0.10.0                     (depends on 1.0.0)
[warn]      +- com.example:lambdas_2.12:0.1.0-SNAPSHOT             (depends on 1.0.0)
[warn]      +- co.fs2:fs2-scodec_2.12:0.10.0                      (depends on 0.10.0)
[warn]      +- org.http4s:jawn-fs2_2.12:0.12.0                    (depends on 0.10.0)
[warn]      +- co.fs2:fs2-io_2.12:0.10.0                          (depends on 0.10.0)
[warn]  * org.typelevel:cats-effect_2.12:1.0.0 is selected over 0.8
[warn]      +- co.fs2:fs2-core_2.12:1.0.0                         (depends on 1.0.0)
[warn]      +- com.example:lambdas_2.12:0.1.0-SNAPSHOT             (depends on 1.0.0)
[warn]      +- org.http4s:http4s-core_2.12:0.18.0                 (depends on 0.8)
[warn]  * org.typelevel:cats-core_2.12:1.4.0 is selected over {1.0.1, 1.3.1}
[warn]      +- io.circe:circe-core_2.12:0.10.0 ()                 (depends on 1.4.0)
[warn]      +- co.fs2:fs2-core_2.12:1.0.0                         (depends on 1.4.0)
[warn]      +- org.typelevel:cats-effect_2.12:1.0.0               (depends on 1.3.1)
[warn]      +- com.example:lambdas_2.12:0.1.0-SNAPSHOT             (depends on 1.3.1)
[warn]      +- org.http4s:http4s-core_2.12:0.18.0                 (depends on 1.0.1)
[warn]  * io.circe:circe-jawn_2.12:0.10.0 is selected over 0.9.1
[warn]      +- io.circe:circe-parser_2.12:0.10.0 ()               (depends on 0.10.0)
[warn]      +- io.circe:circe-fs2_2.12:0.10.0                     (depends on 0.10.0)
[warn]      +- org.http4s:http4s-circe_2.12:0.18.0                (depends on 0.9.1)
[warn]  * org.spire-math:jawn-parser_2.12:0.13.0 is selected over 0.11.0
[warn]      +- io.circe:circe-jawn_2.12:0.10.0 ()                 (depends on 0.13.0)
[warn]      +- org.http4s:jawn-fs2_2.12:0.12.0                    (depends on 0.11.0)
[warn] Run 'evicted' to see detailed eviction warnings
[info] Here are other depedency conflicts that were resolved:
[info]  * commons-logging:commons-logging:1.1.3 is selected over 1.2
[info]      +- com.amazonaws:aws-java-sdk-core:1.11.422           (depends on 1.1.3)
[info]      +- org.apache.httpcomponents:httpclient:4.5.5         (depends on 1.2)
[success] Total time: 2 s, completed Nov 28, 2018 12:56:33 PM

From using sbt-dependency-graph it looks like I will legitimately need two versions of fs2-core: 1.0.0 and 0.10.0.

This is my attempt at using sbt-assembly shading to handle the evictions by renaming the old version of fs2 in the http4s library that needs it. Nothing is actually compiling with this build file so I'm guessing there are multiple problems with the way I have things set up.

//build properties
//sbt.version=1.1.2

//assembly.sbt
//addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.8")

scalaVersion := "2.12.7"
organization := "com.example"

lazy val fatJar = project
  .enablePlugins(AssemblyPlugin)
  .settings(
    libraryDependencies ++= Seq(
      "org.typelevel" %% "cats-core"   % "1.4.0",
      "org.typelevel" %% "cats-effect" % "1.0.0",

      "io.circe" %% "circe-core"    % "0.10.0",
      "io.circe" %% "circe-generic" % "0.10.0",
      "io.circe" %% "circe-parser"  % "0.10.0",
      "io.circe" %% "circe-fs2"     % "0.10.0",

      "co.fs2" %% "fs2-core" % "1.0.0",
      "co.fs2" %% "fs2-io"   % "0.10.0",

      "org.http4s" %% "http4s-circe"  % "0.18.0",
      "org.http4s" %% "http4s-client" % "0.18.0",
      "org.http4s" %% "http4s-core"   % "0.18.0",

      "com.chuusai" %% "shapeless" % "2.3.3",

      "com.amazonaws" % "aws-lambda-java-core" % "1.2.0",
      "com.amazonaws" % "aws-java-sdk-core"    % "1.11.422",
      "com.amazonaws" % "aws-java-sdk-s3"      % "1.11.422",

      "org.scalatest"  %% "scalatest"  % "3.0.5"  % "test",
      "org.scalacheck" %% "scalacheck" % "1.14.0" % "test",

      "io.findify" %% "s3mock" % "0.2.4" % "test"),

    assemblyShadeRules in assembly ++= Seq(
      ShadeRule.rename("co.fs2.**" -> "old_fs2.@1")
        .inLibrary("org.http4s" %% "http4s-circe" % "0.18.0")
        .inLibrary("org.http4s" %% "http4s-core"  % "0.18.0")
    )
  )

lazy val kraken_shaded = (project in file("."))
  .settings(
    name := "kraken-shaded",
    packageBin in Compile := (assembly in (fatJar, Compile)).value

It definitely has breaking changes. I was hoping to figure out how to include both, but shade the usage inside the http4s package so hits fs2 v0.10.0 while the rest of the project uses v1.0.0. — codenoodle, Nov 29 '18 at 16:10
Ahh. Good point. Shading might work if http4s never exposed streams in their interface ....but of course that's not how that package works. I guess my options are to use a different http library until http4s 0.20.0 is out or to downgrade all my other dependencies until they only need fs2 0.10.0. Both are rather unfortunate options. — codenoodle, Nov 29 '18 at 22:45
@LuisMiguelMejíaSuárez if you want to drop an answer summarizing the comments I'll mark it as correct. I appreciate your help. — codenoodle, Dec 10 '18 at 18:54

score 1 · Accepted Answer · answered Dec 10 '18 at 19:38

I'm not an expert in neither http4s nor in fs2, but I believe that the 1.0.x version of fs2 introduces breaking changes.
And, as you can see here, http4s 0.18.x is built against fs2 0.10.x.
Thus, I don't believe you can make them work together.

You may think in shading the 0.10.x version of fs2 for http4s and use the 1.0.x version in your project...
But, I don't think that's even possible, unless you never use a Stream created by http4s nor you pass the Stream to http4s - and that is definitely not way it works.

Nevertheless, you may use http4s 0.20.x.
Right now it is a Milestone, so you can expect changes before it becomes stable - but it might work for now, if you don't want to downgrade to fs2 0.10.x.

sbt shading two versions of a dependency

1 Answers1