4

In my spark application, I'm trying to use fluentd-scala-logger for which I had to include an additional dependency in my build.sbt These are the 2 lines I added in my build.sbt:

resolvers += "Apache Maven Central Repository" at "https://repo.maven.apache.org/maven2/"
"org.fluentd" %% "fluent-logger-scala" % "0.7.0"

My final build.sbt looks like this:

name := "sample"
version := "1.4"
scalaVersion := "2.11.8"
resolvers += "Apache Maven Central Repository" at "https://repo.maven.apache.org/maven2/"
libraryDependencies ++= Seq("org.elasticsearch" %% "elasticsearch-spark" % "2.1.2", "org.apache.spark" %% "spark-sql" % "2.1.2", "org.apache.kafka" % "kafka-clients" % "2.4.1", "org.fluentd" %% "fluent-logger-scala" % "0.7.0")

As soon as I do sbt package to bundle my spark app in a jar, I face below issue:

object tools is not a member of package scala
[error] import scala.tools.nsc.io.File

I didn't face this when my sbt earlier looked like this(without fluentd dependency):

name := "sample"
version := "1.4"
scalaVersion := "2.11.8"
libraryDependencies ++= Seq("org.elasticsearch" %% "elasticsearch-spark" % "2.1.2", "org.apache.spark" %% "spark-sql" % "2.1.2", "org.apache.kafka" % "kafka-clients" % "2.4.1")

Is resolvers line causing the issue? or am I missing something else completely. I'm using sbt version 1.4.5 & Scala version: 2.11.8

Tomer Shetah
  • 8,413
  • 7
  • 27
  • 35
Akshat Chaturvedi
  • 678
  • 1
  • 7
  • 15
  • 1
    That is a pretty weird error, I would recommend you three things. 1) Try with multiple **sbt** versions, like `1.4.4`, `1.3.13`, etc. 2) Asking in the [gitter room](https://gitter.im/sbt/sbt) _(maybe also in the Scala room)_. 3) Search in the **github** issues if it is already reported. – Luis Miguel Mejía Suárez Dec 21 '20 at 14:56
  • Selecting a previous sbt as @LuisMiguelMejíaSuárez suggests, removes the 'object tools' error for me. I'm using intellij, and it overrides the sbt version configured in the project and uses the latest version instead, unless you tell it to use the version from `build.properties`. – Richard Osseweyer Apr 19 '21 at 09:19

1 Answers1

1

I am not sure what causes it, but I found a solution. Please try to add the dependency like this:

libraryDependencies += "org.fluentd" %% "fluent-logger-scala" % "0.7.0" intransitive()

It will import this dependency without its dependencies.

Having that said, I looked at this library dependencies, and tried to exclude all of them 1 by 1:

libraryDependencies += "org.fluentd" %% "fluent-logger-scala" % "0.7.0" excludeAll(
  ExclusionRule("org.msgpack", "msgpack"),
  ExclusionRule("org.slf4j", "slf4j-api"),
  ExclusionRule("ch.qos.logback", "logback-classic"),
  ExclusionRule("junit", "junit"),
  )

but it didn't work. So I really can't explain it.

Tomer Shetah
  • 8,413
  • 7
  • 27
  • 35
  • Would've been great if there could be an explanation. But thanks anyway, my error is gone now. :) – Akshat Chaturvedi Dec 24 '20 at 05:22
  • @AkshatChaturvedi I wish I completely misunderstood what is going on here. I just wanted to know if it works without subdependencies, so those are my tries. I guess that one of the plugins of this package are overriding some namespace, but I am not sure. – Tomer Shetah Dec 24 '20 at 05:43