Unresolved dependencies path SBT - Scala Intellij Project

Question

I have have newly installed and created spark, scala, SBT development environment in intellij but when i am trying to compile SBT, getting unresolved dependencies error.

below is my SBT file

name := "xxxxxxxxxxxxxxxxxxxx"
version := "0.1"
scalaVersion := "2.11.8"

val sparkVersion = "2.3.1"
val jacksonCore = "2.6.7"
val publishMavenStyle = true


resolvers ++= Seq(
  "Artifactory" at "https://binrepo.xxxxxx.com/artifactory/xyz/",
  "Artifactory Common" at "https://binrepo.xxxxxx.com/artifactory/data-engineering-gscl-abc/",
  //"ArtifactorySnapShots" at "https://binrepo.xxxxxx.com/artifactory/xyz/SNAPSHOTS/"
  ("Spark Packages Repo" at "http://dl.bintray.com/spark-packages/maven").withAllowInsecureProtocol(true)
)

libraryDependencies ++= Seq(


  "org.scalamock" %% "scalamock" % "4.4.0" % Test,

  //dependency of xyz-core.
  "com.tgt.dsc.xyz.datapipeline" % "xyz-core_2.11" % "1.8.1",

  //dependency of fields performance specific  common function
  "abc-common" % "abc-common_2.11" % "3.0.0",

  //dependency of reading configuration
  "com.typesafe" % "config" % "1.3.3",

  //spark core libraries, in the production or for spark-submit in local add provided so that dependent jar is not part of assembly jar
  // ex :"org.apache.spark" %% "spark-core" % sparkVersion % provided,
  "org.apache.spark" %% "spark-core" % sparkVersion  % Provided ,
  "org.apache.spark" %% "spark-sql" % sparkVersion  % Provided ,
  "org.apache.spark" %% "spark-hive" % sparkVersion % Provided ,
  "org.scala-lang" % "scala-library" % scalaVersion.value,

  //logging library
  "org.slf4j" % "slf4j-api" % "1.7.29",

  //for doing testing
  "org.scalatest" %% "scalatest" % "3.1.0" % Test,
  "MrPowers" % "spark-fast-tests" % "0.20.0-s_2.11",
  "mrpowers" % "spark-daria" % "0.35.0-s_2.11"
)

enablePlugins(GitVersioning)
assemblyJarName in assembly := s"${name.value}_${scalaVersion.value}-${version.value}.jar"
assemblyMergeStrategy in assembly := {
  //case "META-INF/services/org.apache.spark.sql.sources.DataSourceRegister" => MergeStrategy.concat
  case PathList("META-INF", xs@_*) => MergeStrategy.discard
  case x => MergeStrategy.first
}

//get the token from secrets
val token = sys.env.getOrElse("SONAR_TOKEN", "")

//configurations for sonar integration
sonarProperties ++= Map(
  "sonar.host.url" -> "http://sonarqube.xxxxxx.com:9000",
  "sonar.scala.version" -> "2.11",
  "sonar.projectName" -> "xyz-starter",
  "sonar.projectKey" -> "xyz-starter",
  "sonar.sources" -> "src/main/scala",
  "sonar.tests" -> "src/test/scala",
  "sonar.scala.coverage.reportPaths" -> "xxxxxx/scala-2.11/coverage-report/cobertura.xml,xxxxxx/scala-2.11/scapegoat-report/scapegoat.xml",
  "sonar.login" -> token,
  "sonar.buildbreaker.skip" -> "false"
)

//how much code coverage is needed
coverageMinimum := 80
//fail the build if code coverage is not met
coverageFailOnMinimum := true

Entire sbt file is showing in red including the name, version, scalaVersion

when i compile following is the error which i am getting now

/Library/Java/JavaVirtualMachines/adoptopenjdk-11.jdk/Contents/Home/bin/java -Djline.terminal=jline.UnsupportedTerminal -Dsbt.log.noformat=true -Dfile.encoding=UTF-8 -Didea.managed=true -Dfile.encoding=UTF-8 -jar /Users/xxxxxxx/Library/Application Support/JetBrains/IdeaIC2020.2/plugins/Scala/launcher/sbt-launch.jar
[info] welcome to sbt 1.4.5 (AdoptOpenJDK Java 11.0.9)
[info] loading global plugins from /Users/xxxxxxx/.sbt/1.0/plugins
[info] loading settings for project xxxxxxxxxxxxxxxxxxxxxxxxxx-build from assembly.sbt ...
[info] loading project definition from /Users/xxxxxxx/Documents/GitClone/xxxxxxxxxxxxxxxxxxxxxxxxxx/project
[warn] There may be incompatibilities among your library dependencies; run 'evicted' to see detailed eviction warnings.
[info] loading settings for project xxxxxxxxxxxxxxxxxxxxxxxxxx from build.sbt ...
[info] set current project to xxxxxxxxxxxxxxxxxxxxxxxxxx (in build file:/Users/xxxxxxx/Documents/GitClone/xxxxxxxxxxxxxxxxxxxxxxxxxx/)
[info] sbt server started at local:///Users/xxxxxxx/.sbt/1.0/server/5231834612cf96406db7/sock
[info] started sbt server
sbt:xxxxxxxxxxxxxxxxxxxxxxxxxx>
;set _root_.scala.collection.Seq(historyPath := None,shellPrompt := { _ => "" }
,SettingKey[_root_.scala.Option[_root_.sbt.File]]("sbtStructureOutputFile")
in _root_.sbt.Global := _root_.scala.Some(_root_.sbt.file("/private/var/folders/6p/qvsthwj11q38nxlpn72hv6fr0000gq/T/sbt-structure.xml"))
,SettingKey[_root_.java.lang.String]
("sbtStructureOptions")
in _root_.sbt.Global := "download, resolveClassifiers")
[info] Defining Global / sbtStructureOptions, Global / sbtStructureOutputFile and 1 others.
[info] The new values will be used by cleanKeepGlobs
[info]  Run `last` for details.
[info] Reapplying settings...
[info] set current project to xxxxxxxxxxxxxxxxxxxxxxxxxx (in build file:/Users/xxxxxxx/Documents/GitClone/xxxxxxxxxxxxxxxxxxxxxxxxxx/)
[info] Applying State transformations org.jetbrains.sbt.CreateTasks from /Users/xxxxxxx/Library/Application Support/JetBrains/IdeaIC2020.2/plugins/Scala/repo/org.jetbrains/sbt-structure-extractor/scala_2.12/sbt_1.0/2018.2.1+4-88400d3f/jars/sbt-structure-extractor.jar
[info] Reapplying settings...
[info] set current project to xxxxxxxxxxxxxxxxxxxxxxxxxx (in build file:/Users/xxxxxxx/Documents/GitClone/xxxxxxxxxxxxxxxxxxxxxxxxxx/)
[warn]
[warn]  Note: Unresolved dependencies path:
[error] stack trace is suppressed; run 'last update' for the full output
[error] stack trace is suppressed; run 'last ssExtractDependencies' for the full output
[error] (update) sbt.librarymanagement.ResolveException: Error downloading MrPowers:spark-fast-tests:0.20.0-s_2.11
[error]   Not found
[error]   Not found
[error]   not found: /Users/xxxxxxx/.ivy2/localMrPowers/spark-fast-tests/0.20.0-s_2.11/ivys/ivy.xml
[error]   not found: https://repo1.maven.org/maven2/MrPowers/spark-fast-tests/0.20.0-s_2.11/spark-fast-tests-0.20.0-s_2.11.pom
[error]   not found: https://binrepo.target.com/artifactory/kelsa/MrPowers/spark-fast-tests/0.20.0-s_2.11/spark-fast-tests-0.20.0-s_2.11.pom
[error]   not found: https://binrepo.target.com/artifactory/data-engineering-gscl-fieldperformance/MrPowers/spark-fast-tests/0.20.0-s_2.11/spark-fast-tests-0.20.0-s_2.11.pom
[error]   download error: Caught java.io.IOException: Server returned HTTP response code: 403 for URL: http://dl.bintray.com/spark-packages/maven/MrPowers/spark-fast-tests/0.20.0-s_2.11/spark-fast-tests-0.20.0-s_2.11.pom (Server returned HTTP response code: 403 for URL: http://dl.bintray.com/spark-packages/maven/MrPowers/spark-fast-tests/0.20.0-s_2.11/spark-fast-tests-0.20.0-s_2.11.pom) while downloading http://dl.bintray.com/spark-packages/maven/MrPowers/spark-fast-tests/0.20.0-s_2.11/spark-fast-tests-0.20.0-s_2.11.pom
[error] Error downloading mrpowers:spark-daria:0.35.0-s_2.11
[error]   Not found
[error]   Not found
[error]   not found: /Users/xxxxxxx/.ivy2/localmrpowers/spark-daria/0.35.0-s_2.11/ivys/ivy.xml
[error]   not found: https://repo1.maven.org/maven2/mrpowers/spark-daria/0.35.0-s_2.11/spark-daria-0.35.0-s_2.11.pom
[error]   not found: https://binrepo.target.com/artifactory/kelsa/mrpowers/spark-daria/0.35.0-s_2.11/spark-daria-0.35.0-s_2.11.pom
[error]   not found: https://binrepo.target.com/artifactory/data-engineering-gscl-fieldperformance/mrpowers/spark-daria/0.35.0-s_2.11/spark-daria-0.35.0-s_2.11.pom
[error]   download error: Caught java.io.IOException: Server returned HTTP response code: 403 for URL: http://dl.bintray.com/spark-packages/maven/mrpowers/spark-daria/0.35.0-s_2.11/spark-daria-0.35.0-s_2.11.pom (Server returned HTTP response code: 403 for URL: http://dl.bintray.com/spark-packages/maven/mrpowers/spark-daria/0.35.0-s_2.11/spark-daria-0.35.0-s_2.11.pom) while downloading http://dl.bintray.com/spark-packages/maven/mrpowers/spark-daria/0.35.0-s_2.11/spark-daria-0.35.0-s_2.11.pom
[error] (ssExtractDependencies) sbt.librarymanagement.ResolveException: Error downloading MrPowers:spark-fast-tests:0.20.0-s_2.11
[error]   Not found
[error]   Not found
[error]   not found: /Users/xxxxxxx/.ivy2/localMrPowers/spark-fast-tests/0.20.0-s_2.11/ivys/ivy.xml
[error]   not found: https://repo1.maven.org/maven2/MrPowers/spark-fast-tests/0.20.0-s_2.11/spark-fast-tests-0.20.0-s_2.11.pom
[error]   not found: https://binrepo.target.com/artifactory/kelsa/MrPowers/spark-fast-tests/0.20.0-s_2.11/spark-fast-tests-0.20.0-s_2.11.pom
[error]   not found: https://binrepo.target.com/artifactory/data-engineering-gscl-fieldperformance/MrPowers/spark-fast-tests/0.20.0-s_2.11/spark-fast-tests-0.20.0-s_2.11.pom
[error]   download error: Caught java.io.IOException: Server returned HTTP response code: 403 for URL: http://dl.bintray.com/spark-packages/maven/MrPowers/spark-fast-tests/0.20.0-s_2.11/spark-fast-tests-0.20.0-s_2.11.pom (Server returned HTTP response code: 403 for URL: http://dl.bintray.com/spark-packages/maven/MrPowers/spark-fast-tests/0.20.0-s_2.11/spark-fast-tests-0.20.0-s_2.11.pom) while downloading http://dl.bintray.com/spark-packages/maven/MrPowers/spark-fast-tests/0.20.0-s_2.11/spark-fast-tests-0.20.0-s_2.11.pom
[error] Error downloading mrpowers:spark-daria:0.35.0-s_2.11
[error]   Not found
[error]   Not found
[error]   not found: /Users/xxxxxxx/.ivy2/localmrpowers/spark-daria/0.35.0-s_2.11/ivys/ivy.xml
[error]   not found: https://repo1.maven.org/maven2/mrpowers/spark-daria/0.35.0-s_2.11/spark-daria-0.35.0-s_2.11.pom
[error]   not found: https://binrepo.target.com/artifactory/kelsa/mrpowers/spark-daria/0.35.0-s_2.11/spark-daria-0.35.0-s_2.11.pom
[error]   not found: https://binrepo.target.com/artifactory/data-engineering-gscl-fieldperformance/mrpowers/spark-daria/0.35.0-s_2.11/spark-daria-0.35.0-s_2.11.pom
[error]   download error: Caught java.io.IOException: Server returned HTTP response code: 403 for URL: http://dl.bintray.com/spark-packages/maven/mrpowers/spark-daria/0.35.0-s_2.11/spark-daria-0.35.0-s_2.11.pom (Server returned HTTP response code: 403 for URL: http://dl.bintray.com/spark-packages/maven/mrpowers/spark-daria/0.35.0-s_2.11/spark-daria-0.35.0-s_2.11.pom) while downloading http://dl.bintray.com/spark-packages/maven/mrpowers/spark-daria/0.35.0-s_2.11/spark-daria-0.35.0-s_2.11.pom
[error] Total time: 7 s, completed 19-May-2021, 6:44:44 PM
[info] shutting down sbt server

Any idea on how to resolve it.

Adding the Screenshot of how my console looks

can you post plugins.sbt file content ? – Srinivas May 19 '21 at 14:04 — Srinivas, May 19 '21 at 14:04
not able to find the plugin.sbt file – Ranjeet Paswan May 19 '21 at 15:18 — Ranjeet Paswan, May 19 '21 at 15:18

score 2 · Accepted Answer · answered May 19 '21 at 14:11

Entire sbt file is showing in red including the name, version, scalaVersion

This is likely caused by some missing configuration in IntelliJ, you should have some kind of popup that aks you to "configure Scala SDK". If not, you can go to your module settings and add the Scala SDK.

when i compile following is the error which i am getting now

If you look closely to the error, you should notice this message:

Server returned HTTP response code: 403 for URL: http://dl.bintray.com/spark-packages/maven/MrPowers/spark-fast-tests/0.20.0-s_2.11/spark-fast-tests-0.20.0-s_2.11.pom

The dependencies you are looking for ("MrPowers" % "spark-fast-tests" % "0.20.0-s_2.11" and "mrpowers" % "spark-daria" % "0.35.0-s_2.11") are only available in this repository http://dl.bintray.com/spark-packages/maven/ which seems to require some authentication as the HTTP error code 403 suggests.

Either you can configure authentication, or you can use more recent versions of these libraries that are published on the public Maven central repository:

"com.github.mrpowers" %% "spark-daria" % "0.39.0"
"com.github.mrpowers" %% "spark-fast-tests" % "0.23.0"

EDIT: how did I find that? I used https://mvnrepository.com/search?q=spark-daria to search for your dependencies and found the new ones with the Repository Central flag

Also note there are few things that you may want to change in your build.sbt:

Use the "sbt scheme" to use Scala dependencies without manually setting the Scala version:
- use "com.github.mrpowers" %% "spark-daria" % "0.39.0"
- instead of "com.github.mrpowers" % "spark-daria" % "0.39.0_2.11" (notice the double %% and the absence of suffix _2.11)
Do not declare the Scala library dependency "org.scala-lang" % "scala-library" % scalaVersion.value, this is implicitly provided

For me in intellij there is no option to "configure Scala SDK". I am manually trying it by going to File -> Project Structure -> Under Platform Settings -> Global Libraries i am removing and adding Scala SDK i.e. ivy version 2.12.2 But still while compling the sbt getting the same dependency error. no idea what happening. — Ranjeet Paswan, May 19 '21 at 15:34
The error in IntelliJ could also be related to the missing dependencies, make sure your project compiles fine outside of IntelliJ with `sbt compile` first — Gaël J, May 19 '21 at 15:39
I have added the screenshot of how my Intellij console looks it showing everything in red. — Ranjeet Paswan, May 19 '21 at 15:48
Did you make the suggested changes above to target dependencies available in Maven central repository? — Gaël J, May 19 '21 at 16:07
The changes that it is suggesting is mostly like below Cannot resolve symbol name :1 Cannot resolve symbol := Cannot resolve symbol version Cannot resolve symbol := Not getting what shall I do — Ranjeet Paswan, May 20 '21 at 00:30

score 0 · Answer 2 · answered May 28 '21 at 12:26

The Issue was related to the missing dependency i.e.

"MrPowers" % "spark-fast-tests" % "0.20.0-s_2.11",
"mrpowers" % "spark-daria" % "0.35.0-s_2.11"

After removing this from the code I found that it was also being used in the other dependent JAR

//dependency of fields performance specific  common function
"abc-common" % "abc-common_2.11" % "3.0.0",

Removing from there solved the issue.

Unresolved dependencies path SBT - Scala Intellij Project

2 Answers2