3

I have following SBT file, I am compiling the Scala Code using Apache GraphFrame and also reading the CSV file.

name := "Simple"

version := "1.0"

scalaVersion := "2.10.5"

libraryDependencies ++= Seq(

"org.apache.spark" %% "spark-core" % "1.6.1",

"graphframes" % "graphframes" % "0.2.0-spark1.6-s_2.10",

"org.apache.spark" %% "spark-sql" % "1.0.0",

"com.databricks" % "spark-csv" % "1.0.3"
)

Here is My Code in Scala

import org.graphframes._
import org.apache.spark.sql.DataFrame
    val nodesList = sqlContext.read.format("com.databricks.spark.csv").option("header", "true").option("inferSchema", "true").load("/Users/Desktop/GraphFrame/NodesList.csv")
    val edgesList= sqlContext.read.format("com.databricks.spark.csv").option("header", "true").option("inferSchema", "true").load("/Users/Desktop/GraphFrame/EdgesList.csv")
    val v=nodesList.toDF("id", "name")
    val e=edgesList.toDF("src", "dst", "dist")
    val g = GraphFrame(v, e)

When I try to make the Jar file using SBT, It gives me following error during compiling

[trace] Stack trace suppressed: run last *:update for the full output.
[error] (*:update) sbt.ResolveException: unresolved dependency: graphframes#graphframes;0.2.0-spark1.6-s_2.10: not found
[error] Total time:
Yasir Arfat
  • 645
  • 1
  • 8
  • 21

3 Answers3

6

GraphFrames are not in Maven Central repository yet.

You can:

  1. download artifact on Spark Packages page and install into local repository
  2. Add Spark Packages repository into your SBT build.sbt:

Code in build.sbt:

resolvers += Resolver.url("SparkPackages", url("https://dl.bintray.com/spark-packages/maven/"))
T. Gawęda
  • 15,706
  • 4
  • 46
  • 61
  • Above code is working fine but Now I have same issue for "import org.apache.spark.sql.DataFrame" Error "unresolved dependency: org.apache.spark#spark-sql_2.10_2.10;1.0.0: not found" @T. Gawęda – Yasir Arfat Dec 12 '16 at 14:34
  • @Aroon You've got strange Spark SQL version - "1.0.0" - while Spark Core is 1.6.1. Please change Spark SQL version also to 1.6.1 – T. Gawęda Dec 12 '16 at 14:37
  • @Aroon also doubled version of Scala version suggest that you've changed it to `"org.apache.spark" %% "spark-sql_2.10" % "1.0.0",`. Please use %% or add Scala version. So: `"org.apache.spark" %% "spark-sql" % "1.6.1",` or `"org.apache.spark" % "spark-sql_2.10" % "1.6.1",` – T. Gawęda Dec 12 '16 at 14:38
  • Doesn't work for me. It seems that the package DOES exist on maven. But SBT is unable to resolve it for whatever reason. httpsunresolved dependency: graphframes#graphframes;0.5.0-spark2.1-s_2.11: not found – aclowkay Jun 15 '17 at 07:41
6

For some reason, Resolver.url mentioned in Gawęda's answer was not working for me, the below worked:

resolvers += "SparkPackages" at "https://dl.bintray.com/spark-packages/maven"

libraryDependencies += "graphframes" % "graphframes" % "0.7.0-spark2.4-s_2.11"

Diptanshu
  • 103
  • 2
  • 6
3

I managed to get it working using sbt-spark-package

In project/plugins.sbt, I added:

resolvers += "bintray-spark-packages" at "https://dl.bintray.com/spark-packages/maven/"

addSbtPlugin("org.spark-packages" % "sbt-spark-package" % "0.2.5")

Then, in build.sbt I added:

spDependencies += "graphframes/graphframes:0.5.0-spark2.1-s_2.11"

And it worked.

Hope it helps.

Javier Alba
  • 381
  • 1
  • 3
  • 11
  • This works for me. Just note that if you're going to use the `sbt-spark-package`, then make sure you use `sbt 0.13.16`, as that package does not work with `sbt 1.1.1` (see https://github.com/databricks/sbt-spark-package/issues/35) – Shafique Jamal Apr 05 '18 at 04:27