3

I would like to use the hbase hortonworks connector. github guide

But I don't know how to import it in my project. I have the following build.sbt :

name := "project"

version := "1.0"

scalaVersion := "2.11.8"

libraryDependencies ++= Seq(
  "org.apache.spark" % "spark-core_2.11" % "2.2.0",
  "org.apache.spark" % "spark-sql_2.11" % "2.2.0",
  "org.scala-lang" % "scala-compiler" % "2.11.8",
  "com.hortonworks" % "shc" % "1.1.2-2.1-s_2.11-SNAPSHOT"
)

And it gives me the follwing unresolved dependencies :

Error:Error while importing SBT project:
...
[warn] ==== local: tried [warn] ivy.xml [warn] ==== public: tried [warn]
https://repo1.maven.org/maven2/com/hortonworks/shc/1.1.2-2.1-s_2.11-SNAPSHOT/shc-1.1.2-2.1-s_2.11-SNAPSHOT.pom [info] Resolving jline#jline;2.12.1 ... [warn] :::::::::::::::::::::::::::::::::::::::::::::: [warn] ::
UNRESOLVED DEPENDENCIES :: [warn] :::::::::::::::::::::::::::::::::::::::::::::: [warn] :: com.hortonworks#shc;1.1.2-2.1-s_2.11-SNAPSHOT: not found [warn] :::::::::::::::::::::::::::::::::::::::::::::: [warn] [warn] Note: Unresolved dependencies path: [warn] com.hortonworks:shc:1.1.2-2.1-s_2.11-SNAPSHOT (build.sbt#L8-14) [warn] +- default:project_2.11:1.0 [trace] Stack trace suppressed: run 'last *:ssExtractDependencies' for the full output. [trace] Stack trace suppressed: run 'last :update' for the full output. [error] (:ssExtractDependencies) sbt.ResolveException: unresolved dependency: com.hortonworks#shc;1.1.2-2.1-s_2.11-SNAPSHOT: not found [error] (*:update) sbt.ResolveException: unresolved dependency: com.hortonworks#shc;1.1.2-2.1-s_2.11-SNAPSHOT: not found [error] Total time: 7 s, completed 3 août 2017 11:36:37

Omegaspard
  • 1,828
  • 2
  • 24
  • 52

2 Answers2

6

In your solution you need to add the resolvers line which represents the repository.

scalaVersion := "2.11.12"

val sparkVersion = "2.1.0"

resolvers += "Hortonworks Repository" at "http://repo.hortonworks.com/content/repositories/releases/"

libraryDependencies += "org.apache.spark" %% "spark-core" % sparkVersion
libraryDependencies += "org.apache.spark" %% "spark-sql" % sparkVersion

libraryDependencies ++= Seq(
  "com.hortonworks" % "shc-core" % "1.1.1-2.1-s_2.11"
)
cipri.l
  • 819
  • 10
  • 22
0

My recommendation is not to go with the most latest version of the Spark because all latest versions have some bugs to be fixed. Try to use 2.1.2 spark version instead of 2.2.0 and go to the maven repository try this link

Here you'll get all related dependencies for spark.

Shashank Kumar
  • 21
  • 3