0

When I run my hadoop mapreduce word count jar in hadoop folder in shell, it is running properly and the output is generated correctly,

Since I use yarn in case of hadoop 2.4.1, when I run from eclipse for MapReduce Sample program, MAP process completed and getting failed in reduce process.

Its clear that the problem is with jar configuration.

Please find the jars, I have added...

enter image description here

This is the error I got

INFO: reduce task executor complete. Nov 21, 2014 8:50:35 PM org.apache.hadoop.mapred.LocalJobRunner$Job run WARNING: job_local1638918104_0001 java.lang.Exception: java.lang.NoSuchMethodError: org.apache.hadoop.mapred.ReduceTask.setLocalMapFiles(Ljava/util/Map;)V at org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:462) at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:529) Caused by: java.lang.NoSuchMethodError: org.apache.hadoop.mapred.ReduceTask.setLocalMapFiles(Ljava/util/Map;)V at org.apache.hadoop.mapred.LocalJobRunner$Job$ReduceTaskRunnable.run(LocalJobRunner.java:309) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:722)

Exception in thread "Thread-12" java.lang.NoClassDefFoundError: org/apache/commons/httpclient/HttpMethod at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:562) Caused by: java.lang.ClassNotFoundException: org.apache.commons.httpclient.HttpMethod at java.net.URLClassLoader$1.run(URLClassLoader.java:366) at java.net.URLClassLoader$1.run(URLClassLoader.java:355) at java.security.AccessController.doPrivileged(Native Method) at java.net.URLClassLoader.findClass(URLClassLoader.java:354) at java.lang.ClassLoader.loadClass(ClassLoader.java:423) at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) at java.lang.ClassLoader.loadClass(ClassLoader.java:356) ... 1 more

VC1
  • 1,660
  • 4
  • 25
  • 42
Harry
  • 3,072
  • 6
  • 43
  • 100

1 Answers1

0

As per the screenshot, you are manually adding all the dependent jars to the classpath. It's highly recommended to use maven for this, which will automate the process of adding dependent jars to the classpath. We just need to add main dependent jars.
I used the following dependencies in pom.xml which helped me to run without any issues.. 

<properties>
    <hadoop.version>2.5.2</hadoop.version>
</properties>

<dependencies>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-hdfs</artifactId>
            <version>${hadoop.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-common</artifactId>
            <version>${hadoop.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-client</artifactId>
            <version>${hadoop.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-mapreduce-client-core</artifactId>
            <version>${hadoop.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-yarn-api</artifactId>
            <version>${hadoop.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-yarn-common</artifactId>
            <version>${hadoop.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-auth</artifactId>
            <version>${hadoop.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-yarn-server-nodemanager</artifactId>
            <version>${hadoop.version}</version>
        </dependency>
        <dependency>
            <groupId>org.apache.hadoop</groupId>
            <artifactId>hadoop-yarn-server-resourcemanager</artifactId>
            <version>${hadoop.version}</version>
        </dependency>
    </dependencies>

come to your problem, I checked in the classpath, there are exactly 82 jar files available.
It will be tedious job to find each jar like this.
You can add the functional wise jars HERE.
Other workaround would be, add all the jar files in installed hadoop directory path as <hadoop-installed>/share/hadoop/ and add all jars from all the lib folder. which is the best thing you can do.. or
Add only avro specific jars, because exception thrown by avro class as per the screenshot. This could solve avro jars issue. but you may face other dependecy issues. I also faced the same problem while working with Hadoop V1. So later i realized and using Maven with Hadoop V2. So no worries of dependent jars.
Your focus will be on Hadoop and Business needs. :)
Hope it helps you..

Mr.Chowdary
  • 3,389
  • 9
  • 42
  • 66