0

For the project, I need ignite-spark dependency to be added, but adding the below line and Sync is giving error message Modules were resolved with conflicting cross-version suffixes in ProjectRef.

libraryDependencies += "org.apache.ignite" % "ignite-spark_2.10" % "2.3.0"

Also tried

libraryDependencies += "org.apache.ignite" %% "ignite-spark" % "2.3.0"

Scala version: 2.11.12 Spark:2.3.0 Ignite: 2.10

build.sbt

name := "ScalaProjectDemo2.11.12"

version := "0.1"

scalaVersion := "2.11.12"
libraryDependencies += "org.apache.spark" %% "spark-sql" % "2.3.0" % "provided"
libraryDependencies += "org.apache.ignite" % "ignite-spring" % "2.10.0"
libraryDependencies += "org.apache.ignite" % "ignite-indexing" % "2.10.0"
libraryDependencies += "org.apache.ignite" % "ignite-core" % "2.10.0"
libraryDependencies += "org.apache.spark" %% "spark-core" % "2.3.0"
libraryDependencies += "org.apache.ignite" % "ignite-spark_2.10" % "2.3.0"

D:/Java/jdk1.8.0_301/bin/java.exe -Djline.terminal=jline.UnsupportedTerminal -Dsbt.log.noformat=true -Dfile.encoding=UTF-8 -Didea.managed=true -Dfile.encoding=UTF-8 -jar C:/Users/admin/AppData/Roaming/JetBrains/IdeaIC2020.1/plugins/Scala/launcher/sbt-launch.jar
[info] Loading global plugins from C:\Users\admin\.sbt\1.0\plugins
[info] Loading project definition from C:\Users\admin\IdeaProjects\ScalaProjectDemo2.11.12\project
[info] Loading settings for project scalaprojectdemo2-11-12 from build.sbt ...
[info] Set current project to ScalaProjectDemo2.11.12 (in build file:/C:/Users/admin/IdeaProjects/ScalaProjectDemo2.11.12/)
[info] sbt server started at local:sbt-server-800963285a52579c1956
sbt:ScalaProjectDemo2.11.12>
[info] Defining Global / sbtStructureOptions, Global / sbtStructureOutputFile and 1 others.
[info] The new values will be used by cleanKeepGlobs
[info]  Run `last` for details.
[info] Reapplying settings...
[info] Set current project to ScalaProjectDemo2.11.12 (in build file:/C:/Users/admin/IdeaProjects/ScalaProjectDemo2.11.12/)
[info] Applying State transformations org.jetbrains.sbt.CreateTasks from C:/Users/admin/AppData/Roaming/JetBrains/IdeaIC2020.1/plugins/Scala/repo/org.jetbrains/sbt-structure-extractor/scala_2.12/sbt_1.0/2018.2.1+4-88400d3f/jars/sbt-structure-extractor.jar
[info] Reapplying settings...
[info] Set current project to ScalaProjectDemo2.11.12 (in build file:/C:/Users/admin/IdeaProjects/ScalaProjectDemo2.11.12/)
[warn] There may be incompatibilities among your library dependencies; run 'evicted' to see detailed eviction warnings.
[error] Modules were resolved with conflicting cross-version suffixes in ProjectRef(uri("file:/C:/Users/admin/IdeaProjects/ScalaProjectDemo2.11.12/"), "scalaprojectdemo2-11-12"):
[error]    org.apache.spark:spark-launcher _2.10, _2.11
[error]    org.apache.spark:spark-sketch _2.10, _2.11
[error]    org.json4s:json4s-ast _2.10, _2.11
[error]    org.apache.spark:spark-catalyst _2.10, _2.11
[error]    org.apache.spark:spark-network-shuffle _2.10, _2.11
[error]    com.twitter:chill _2.10, _2.11
[error]    org.apache.spark:spark-sql _2.11, _2.10
[error]    org.json4s:json4s-jackson _2.11, _2.10
[error]    com.fasterxml.jackson.module:jackson-module-scala _2.10, _2.11
[error]    org.json4s:json4s-core _2.10, _2.11
[error]    org.apache.spark:spark-unsafe _2.11, _2.10
[error]    org.apache.spark:spark-tags _2.11, _2.10
[error]    org.apache.spark:spark-core _2.10, _2.11
[error]    org.apache.spark:spark-network-common _2.10, _2.11
[error] stack trace is suppressed; run 'last update' for the full output
[error] stack trace is suppressed; run 'last ssExtractDependencies' for the full output
[error] (update) Conflicting cross-version suffixes in: org.apache.spark:spark-launcher, org.apache.spark:spark-sketch, org.json4s:json4s-ast, org.apache.spark:spark-catalyst, org.apache.spark:spark-network-shuffle, com.twitter:chill, org.apache.spark:spark-sql, org.json4s:json4s-jackson, com.fasterxml.jackson.module:jackson-module-scala, org.json4s:json4s-core, org.apache.spark:spark-unsafe, org.apache.spark:spark-tags, org.apache.spark:spark-core, org.apache.spark:spark-network-common
[error] (ssExtractDependencies) Conflicting cross-version suffixes in: org.apache.spark:spark-launcher, org.apache.spark:spark-sketch, org.json4s:json4s-ast, org.apache.spark:spark-catalyst, org.apache.spark:spark-network-shuffle, com.twitter:chill, org.apache.spark:spark-sql, org.json4s:json4s-jackson, com.fasterxml.jackson.module:jackson-module-scala, org.json4s:json4s-core, org.apache.spark:spark-unsafe, org.apache.spark:spark-tags, org.apache.spark:spark-core, org.apache.spark:spark-network-common
[error] Total time: 9 s, completed Sep 18, 2021 6:11:55 AM
[info] shutting down sbt server

Please provide your suggestions.

Thanks in advance.

Smera
  • 13
  • 4
  • 1
    What is the error you get when you try `"org.apache.ignite" %% "ignite-spark" % "2.3.0"`? – Jasper-M Sep 17 '21 at 12:56
  • Error found with above dependancy is - [error] (update) sbt.librarymanagement.ResolveException: Error downloading org.apache.ignite:ignite-spark_2.11:2.3.0. Issue is now fixed after using 2.10.0 instead of 2.3.0. – Smera Sep 17 '21 at 17:52

1 Answers1

1

Looking at Maven Repository.

We can see that the 2.3.0 of ignite-spark only supports Scala 2.10 (and thus also depends on older versions of Spark).

You may want to upgrade to at least 2.7.6 which (only) supports Scala 2.11 and is based on Spark 2.3; which is the same version you were using.

Luis Miguel Mejía Suárez
  • 21,686
  • 3
  • 28
  • 45
  • Thanks much for your response. Using 2.10.0 fixed the error ---- libraryDependencies += "org.apache.ignite" % "ignite-spark" % "2.10.0" – Smera Sep 17 '21 at 17:47