0

I have run a command spark-shell --jars /home/krishnamahi/graphframes-0.4.0-spark2.1-s_2.11.jar and it threw me an error

error: missing or invalid dependency detected while loading class file 'Logging.class'. Could not access term typesafe in package com, because it (or its dependencies) are missing. Check your build definition for missing or conflicting dependencies. (Re-run with -Ylog-classpath to see the problematic classpath.) A full rebuild may help if 'Logging.class' was compiled against an incompatible version of com. error: missing or invalid dependency detected while loading class file 'Logging.class'. Could not access term scalalogging in value com.typesafe, because it (or its dependencies) are missing. Check your build definition for missing or conflicting dependencies. (Re-run with -Ylog-classpath to see the problematic classpath.) A full rebuild may help if 'Logging.class' was compiled against an incompatible version of com.typesafe. error: missing or invalid dependency detected while loading class file 'Logging.class'. Could not access type LazyLogging in value com.slf4j, because it (or its dependencies) are missing. Check your build definition for missing or conflicting dependencies. (Re-run with -Ylog-classpath to see the problematic classpath.) A full rebuild may help if 'Logging.class' was compiled against an incompatible version of com.slf4j.

I am using Spark Version 2.1.1, Scala Version 2.11.8, JDK Version 1.8.0_131, CentOS7 64-bit, Hadoop 2.8.0. Can anyone please tell me what additional command should I give for perfect run of program? Thanks in advance.

Jacek Laskowski
  • 72,696
  • 27
  • 242
  • 420
D Sai Krishna
  • 178
  • 1
  • 13

2 Answers2

1

If you want to play with GraphFrames use --packages command-line option of spark-shell instead.

--packages Comma-separated list of maven coordinates of jars to include on the driver and executor classpaths. Will search the local maven repo, then maven central and any additional remote repositories given by --repositories. The format for the coordinates should be groupId:artifactId:version.

For graphframes-0.4.0-spark2.1-s_2.11.jar that'd be as follows:

$SPARK_HOME/bin/spark-shell --packages graphframes:graphframes:0.4.0-spark2.1-s_2.11

which I copied verbatim from How to section of GraphFrames project.

That way you don't have to search for all the (transitive) dependencies of GraphFrames library as Spark will do it for you automatically.

Jacek Laskowski
  • 72,696
  • 27
  • 242
  • 420
0

I have installed raw Hadoop, with all components Hive, Pig, Spark of latest versions. It then worked for me. I used Cent OS 7. Order for installing Hadoop with components is

  1. Anaconda3/Python3 (Since Spark 2.x doesn't support Python 2)
  2. Hadoop
  3. Pig
  4. Hive
  5. Hbase
  6. Spark

All the components should be in a single go, in same terminal. After Spark installation, restart the system.

D Sai Krishna
  • 178
  • 1
  • 13