0

Please help to run Scalding tutorial. I have Hadoop 2.2 running on a single node and trying to run Scalding tutorial: https://github.com/Cascading/scalding-tutorial/ After successfuly buiding 'fat jar' with these commands:

$ git clone git://github.com/Cascading/scalding-tutorial.git
$ cd scalding-tutorial
$ sbt assembly

I try to run tutorial examples as suggested with this command:

$ yarn jar target/scalding-tutorial-0.8.11.jar <TutorialPart> --local <addtional arguments>

Both --local and --hdfs fail with java.lang.ClassNotFoundException:

$ yarn jar target/scala-2.9.3/scalding-assembly-0.10.0.jar 1 --local
Exception in thread "main" java.lang.ClassNotFoundException: 1
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.hadoop.util.RunJar.main(RunJar.java:205)

$ yarn jar target/scala-2.9.3/scalding-assembly-0.10.0.jar 1 --hdfs
Exception in thread "main" java.lang.ClassNotFoundException: 1
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.hadoop.util.RunJar.main(RunJar.java:205)

Update

Changing command argument to 'Tutorial1', 'Tutorial0' does not help either:

$ yarn jar target/scala-2.9.3/scalding-assembly-0.10.0.jar Tutorial1 --local
Exception in thread "main" java.lang.ClassNotFoundException: Tutorial1
 at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
 at java.security.AccessController.doPrivileged(Native Method)
 at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
 at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
 at java.lang.Class.forName0(Native Method)
 at java.lang.Class.forName(Class.java:247)
 at org.apache.hadoop.util.RunJar.main(RunJar.java:205)

$ yarn jar target/scala-2.9.3/scalding-assembly-0.10.0.jar Tutorial0 --local
Exception in thread "main" java.lang.ClassNotFoundException: Tutorial0
at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:247)
at org.apache.hadoop.util.RunJar.main(RunJar.java:205)
DarqMoth
  • 603
  • 1
  • 13
  • 31

1 Answers1

0

You are passing a wrong name for the main class, that is why it can't find it. It should be Tutorial1 instead of just 1. You can see the error in the stack trace:

Exception in thread "main" java.lang.ClassNotFoundException: 1

There is no class called 1. Try:

$ yarn jar target/scala-2.9.3/scalding-assembly-0.10.0.jar Tutorial1 --local

EDIT: it works just fine to me with this command:

$ yarn jar target/scalding-tutorial-0.8.11.jar Tutorial1 --local
Balduz
  • 3,560
  • 19
  • 35
  • Thanks, makes sense! Unfortunately changing command argument to 'Tutorial1', 'Tutorial0' does not help either, please see 'update' in my question. – DarqMoth Jul 02 '14 at 13:48
  • 1
    I have just followed the steps of the tutorial and it did work for me... I have noticed one thing after going through them, the jar should be located in `target/scalding-tutorial-0.8.11.jar`, not in `target/scala-2.9.3/scalding-assembly-0.10.0.jar`. Could this be the error? I have edited my post with the exact command I am using. – Balduz Jul 03 '14 at 07:49
  • Not in my case, still have the same error, no matter where I put this jar. Another idea: I do not have JAVA_HOME and JAVA_CLASSPATH set, may this be the case? – DarqMoth Jul 03 '14 at 08:11
  • 1
    The error here is that it cannot find the class you are specifying, so that should not be the problem... But what I find odd is that the jar generated in your case has a different name and is in a different folder. Could you delete it and call `sbt assembly` again? You said "no matter where I put this jar", but it's not only the location, in your command the jar name was different. – Balduz Jul 03 '14 at 08:13
  • But in my case `sbt assembly`(after `sbt update clean`) results in a single `scalding/target/scala-2.9.3/scalding-assembly-0.10.0.jar` file built. Why it does not build `target/scalding-tutorial-0.8.11.jar`? How to build it? – DarqMoth Jul 03 '14 at 09:01
  • Solved! When one follows tutorial instructions at https://github.com/Cascading/scalding-tutorial/ and does 'git clone git://github.com/Cascading/scalding-tutorial.git` instead of tutorial one gets a complete `Scalding` project. Running then `sbt assembly` creates fat Scalding jar `scalding/target/scala-2.9.3/scalding-assembly-0.10.0.jar` instead of tutorial jar. To buld and run tutorial one needs to download a *zip* file from the same git page. This zip extarcts to `scalding-tutorial-wip-2.5`. After `sbt assembly` in this dir,`yarn jar target/scalding-tutorial-0.8.11.jar Tutorial1` works. – DarqMoth Jul 03 '14 at 09:55
  • Glad you solved it! I didn't need to download a zip file, I did the git clone alone and it worked... It's odd, but you made it! :) Please accept the answer if it helped you! – Balduz Jul 03 '14 at 10:00