GraphFrame: missing or invalid dependency detected while loading class file

Question

I am trying to create a graph using spark graphframe

here is the code:

import org.graphframes._

// Node DataFrames
val v = sqlContext.createDataFrame(List(
  ("a", "Alice", 34),
  ("b", "Bob", 36),
  ("c", "Charlie", 30),
  ("d", "David", 29),
  ("e", "Esther", 32),
  ("f", "Fanny", 36),
  ("g", "Gabby", 60)
)).toDF("id", "name", "age")

// Edge DataFrame
val e = sqlContext.createDataFrame(List(
  ("a", "b", "friend"),
  ("b", "c", "follow"),
  ("c", "b", "follow"),
  ("f", "c", "follow"),
  ("e", "f", "follow"),
  ("e", "d", "friend"),
  ("d", "a", "friend"),
  ("a", "e", "friend")
)).toDF("src", "dst", "relationship")

// Create a GraphFrame
val g = GraphFrame(v, e)

But this is the error I am getting:

error: missing or invalid dependency detected while loading class file 'GraphFrame.class'. Could not access type Logging in package org.apache.spark, because it (or its dependencies) are missing. Check your build definition for missing or conflicting dependencies. (Re-run with -Ylog-classpath to see the problematic classpath.) A full rebuild may help if 'GraphFrame.class' was compiled against an incompatible version of org.apache.spark.

I am using Apache Spark 2.1 and Scala 2.11. Any suggestion what can be the issue?

score 0 · Answer 1 · answered Sep 11 '17 at 12:02

Download following packages from maven central repo

com.typesafe.scala-logging_scala-logging-api_2.11-2.1.2.jar    
graphframes_graphframes-0.5.0-spark2.1-s_2.11.jar  
org.slf4j_slf4j-api-1.7.7.jar
com.typesafe.scala-logging_scala-logging-slf4j_2.11-2.1.2.jar  
org.scala-lang_scala-reflect-2.11.0.jar

Add the following to your sparks-default.conf file(comma separated list of absolute path where the above downloaded jars are located)

spark.jars            path_2_jar/org.slf4j_slf4j-api-1.7.7.jar, path_2_jar/org.scala-lang_scala-reflect-2.11.0.jar, path_2_jar/graphframes_graphframes-0.5.0-spark2.1-s_2.11.jar, path_2_jar/com.typesafe.scala-logging_scala-logging-slf4j_2.11-2.1.2.jar, path_2_jar/com.typesafe.scala-logging_scala-logging-api_2.11-2.1.2.jar

GraphFrame: missing or invalid dependency detected while loading class file

1 Answers1