Hadoop 2.3.0 wordcount runs forever

Question

I am trying to test my hadoop installation by running a wordcount job. My problem is that the job gets stuck at ACCEPTED state and seems to run forever. I am using hadoop 2.3.0 and tried fix the problem by following an answer to this question here but it didn't work for me.

This is what I have:

C:\hadoop-2.3.0>yarn jar share/hadoop/mapreduce/hadoop-mapreduce-examples-2.3.0.jar wordcount /data/test.txt /data/output
15/03/15 15:36:07 INFO client.RMProxy: Connecting to ResourceManager at/0.0.0.0:8032
15/03/15 15:36:09 INFO input.FileInputFormat: Total input paths to process : 1
15/03/15 15:36:10 INFO mapreduce.JobSubmitter: number of splits:1
15/03/15 15:36:10 INFO mapreduce.JobSubmitter: Submitting tokens for job: job_14 26430101974_0001
15/03/15 15:36:11 INFO impl.YarnClientImpl: Submitted application application_14 26430101974_0001
15/03/15 15:36:11 INFO mapreduce.Job: The url to track the job: http://Agata-PC:8088/proxy/application_1426430101974_0001/
15/03/15 15:36:11 INFO mapreduce.Job: Running job: job_1426430101974_0001

This is my mapred-site.xml:

<configuration>
<property>
   <name>mapreduce.framework.name</name>
   <value>yarn</value>
</property>
 <property>
    <name>mapred.job.tracker</name>
    <value>127.0.0.1:9001</value>
</property>
   <property>
    <name>mapreduce.jobtracker.staging.root.dir</name>
    <value>/user</value>
</property>
<property>
    <name>mapreduce.history.server.http.address</name>
    <value>127.0.0.1:51111</value>
    <description>Http address of the history server</description>
    <final>false</final>
</property>
<property>
    <name>yarn.app.mapreduce.am.resource.mb</name>
    <value>1024</value>
</property>
<property>
    <name>yarn.app.mapreduce.am.command-opts</name>
    <value>-Xmx768m</value>
</property>
<property>
    <name>mapreduce.map.cpu.vcores</name>
    <value>1</value>
    <description>The number of virtual cores required for each map task.</description>
</property>
<property>
    <name>mapreduce.reduce.cpu.vcores</name>
    <value>1</value>
    <description>The number of virtual cores required for each map task.</description>
</property>
<property>
    <name>mapreduce.map.memory.mb</name>
    <value>1024</value>
    <description>Larger resource limit for maps.</description>
</property>
<property>
    <name>mapreduce.map.java.opts</name>
    <value>-Xmx768m</value>
    <description>Heap-size for child jvms of maps.</description>
</property>
<property>
    <name>mapreduce.reduce.memory.mb</name>
    <value>1024</value>
    <description>Larger resource limit for reduces.</description>
</property>
<property>
    <name>mapreduce.reduce.java.opts</name>
    <value>-Xmx768m</value>
    <description>Heap-size for child jvms of reduces.</description>
</property>
</configuration>

And this is my yarn-site.xml:

<configuration>
    <property>
        <name>yarn.scheduler.minimum-allocation-mb</name>
        <value>128</value>
        <description>Minimum limit of memory to allocate to each container request at the Resource Manager.</description>
    </property>
    <property>
        <name>yarn.scheduler.minimum-allocation-vcores</name>
        <value>1</value>
        <description>The minimum allocation for every container request at the RM, in terms of virtual CPU cores. Requests lower than this won't take effect, and the specified value will get allocated the minimum.</description>
    </property>
    <property>
        <name>yarn.scheduler.maximum-allocation-vcores</name>
        <value>2</value>
        <description>The maximum allocation for every container request at the RM, in terms of virtual CPU cores. Requests higher than this won't take effect, and will get capped to this value.</description>
    </property>
    <property>
        <name>yarn.nodemanager.resource.memory-mb</name>
        <value>2048</value>
        <description>Physical memory, in MB, to be made available to running containers</description>
    </property>
    <property>
        <name>yarn.nodemanager.resource.cpu-vcores</name>
        <value>4</value>
        <description>Number of CPU cores that can be allocated for containers.</description>
    </property>
</configuration>

Any help is much appreciated.

possible duplicate of [Wordcount program is stuck in hadoop-2.3.0](http://stackoverflow.com/questions/23397763/wordcount-program-is-stuck-in-hadoop-2-3-0) — Shash, Jun 07 '15 at 14:59

score 1 · Answer 1 · answered Mar 15 '15 at 18:22

1

Did you try restarting your hadoop's processes or clusters? There might be some works still running.

May be it will be helpful to see the log by following the url of the job or by going through the hadoop url.

Cheers.

answered Mar 15 '15 at 18:22

Spacez

77
2
11

Yes, I tried to run the job a couple of times and it keeps getting stuck there. – ak. Mar 15 '15 at 18:34
Can you please show me the logs of the jobs you're running, including the result from jps, just in case. – Spacez Mar 15 '15 at 18:37
This is what I can see in the overview. `User: Agata Name: word count Application Type: MAPREDUCE State: ACCEPTED FinalStatus: UNDEFINED Started: 15-Mar-2015 20:23:31 Elapsed: 14mins, 33sec Tracking URL: UNASSIGNED Diagnostics:` – ak. Mar 15 '15 at 19:40
On the left side of the page, checkout the Tools, then local logs, /userlogs, and find the jobs name that match yours. If possible, you may need to copy the content from every container inside the jobs logs. Also, please run the 'jps' command in the terminal and copy the result. – Spacez Mar 15 '15 at 19:47
In local logs there is nothing except one authentication link that leads to a blank page :( – ak. Mar 15 '15 at 19:52
Did you copy from local to HDFS? I mean, is the path "/data/test.txt" really exist. Try "yarn dfs -ls /data" (sry if it's mistyped, i'm not sure about the exact command). Also, please run the jps command like this http://puu.sh/gBUf9/2e57c5b9f0.jpg I would like to see the result from both jps and dfs -ls – Spacez Mar 15 '15 at 22:28

score 0 · Answer 2 · answered Mar 16 '15 at 02:39

0

I have run into similar issue earlier, you might have a infinite loop in mapper or reducer . Check if your reducer is properly handling iterable.

answered Mar 16 '15 at 02:39

Karthik

1,801
1
13
21

Hadoop 2.3.0 wordcount runs forever

2 Answers2