1

I'm basicallly following the guide on https://github.com/amplab/shark/wiki/Running-Shark-Locally. I downloaded scala I'm using ec2 amazon linux

my shark/shark-0.8.0/conf/shark-env.sh configuration file look like this

export SPARK_MEM=1g
export SHARK_MASTER_MEM=1g
export SCALA_HOME="/home/user2/scala/"
export HIVE_HOME="/home/user2/shark/hive-0.9.0-shark-0.8.0-bin/"

I also have JAVA_HOME set to /usr/lib/jvm/jre

when i type java -version it returns:

java version "1.7.0_55"
OpenJDK Runtime Environment (amzn-2.4.7.1.40.amzn1-x86_64 u55-b13)
OpenJDK 64-Bit Server VM (build 24.51-b03, mixed mode)

Every time trying to run shark with the following:

shark/shark-0.8.0/bin/shark

I don't understand why I'm getting this bunch of error. DOes anybody know??

Exception in thread "main" java.lang.NoClassDefFoundError: scala/ScalaObject
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:482)
Caused by: java.lang.ClassNotFoundException: scala.ScalaObject
John Petrone
  • 26,943
  • 6
  • 63
  • 68
user2773013
  • 3,102
  • 8
  • 38
  • 58

2 Answers2

0

Two issues that could be causing your problems:

  1. First of all that first error message indicates that it cannot find Scala where it expects to see it. I would double check that you can run scala and sbt from the command line from that directory. I believe SCALA_HOME has to point to the top level directory and not another one containing Scala, so normally this would be /scala-2.9.3

  2. Second, and this is just a possible issue, you need to make certain you've got Scala version 2.9.3 - Shark does not currently run on newer versions and I believe you will get similar (but different) error messages if you try it. Most links out there are for the generic scala download page - here's the 2.9.3 one: http://www.scala-lang.org/download/2.9.3.html

A review of this writeup may be helpful:

http://cornercases.wordpress.com/2013/08/07/getting-the-spark-demo-to-work/

John Petrone
  • 26,943
  • 6
  • 63
  • 68
0

Right Scala library version is important: I am using Spark 1.4.0, it supports Scala 2.10.5. The 2.10.4 library does not work.

Soumya Kanti
  • 1,429
  • 1
  • 17
  • 28