More than 120 counters in hadoop

Question

There's a limit for Hadoop counter size. It's 120 by default. I try to use the configuration "mapreduce.job.counters.limit" to change that, but it doesn't work. I've seen the source code. It's like the instance of JobConf in class "org.apache.hadoop.mapred.Counters" is private. Have anybody seen that before? What's your solution? THX :)

I haven't run into the same problem (mainly since I haven't used counters much), but I'm using 1.0.3 and the JobConf stuff is different. Try Lorand's solution below, it looks like that will work. — HypnoticSheep, Aug 28 '12 at 18:28

score 8 · Accepted Answer · answered Aug 28 '12 at 09:51

8

You can override that property in mapred-site.xml on your JT, TT, client nodes but make sure that this will be a system-wide modification:

<configuration>
  ...
  <property>
    <name>mapreduce.job.counters.limit</name>
    <value>500</value>
  </property>
  ...
</configuration>

Then restart the mapreduce service on your cluster.

answered Aug 28 '12 at 09:51

Lorand Bendig

10,630
1
38
45

I tried your solution. It works. Thx :). I just wonder if there is a way to solve this problem dynamically. – size.of.world Aug 29 '12 at 03:48
Is there no way to do this for a single job? – javamonkey79 Jan 10 '15 at 02:06

Dima · Answer 2 · 2016-09-21T21:24:24.303

6

In Hadoop 2, this configuration parameter is called

mapreduce.job.counters.max

Setting it on the command line or in your Configuration object isn't enough, though. You need to call the static method

org.apache.hadoop.mapreduce.counters.Limits.init()

in the setup() method of your mapper or reducer to get the setting to take effect.

Tested with 2.6.0 and 2.7.1.

edited Sep 21 '16 at 21:24

answered Nov 13 '15 at 20:20

Dima

2,601
1
18
9

score 1 · Answer 3 · answered Mar 01 '16 at 06:43

1

The para is set by config file, while paras below will take effect

mapreduce.job.counters.max=1000 
mapreduce.job.counters.groups.max=500 
mapreduce.job.counters.group.name.max=1000 
mapreduce.job.counters.counter.name.max=500

answered Mar 01 '16 at 06:43

yanghaogn

833
7
15

score 0 · Answer 4 · answered Oct 21 '16 at 13:42

Just adding this in case anyone else faces the same problem we did: increasing the counters from with MRJob.

To raise the number of counters, add emr_configurations to your mrjob.conf (or pass it to MRJob as a config parameter):

runners:
  emr:
    emr_configurations:
      - Classification: mapred-site
        Properties:
          mapreduce.job.counters.max: 1024
          mapreduce.job.counters.counter.name.max: 256
          mapreduce.job.counters.groups.max: 256
          mapreduce.job.counters.group.name.max: 256

score -1 · Answer 5 · edited Oct 13 '15 at 14:22

-1

We can customize the limits as command line options only for specific jobs, instead of making change in mapred-site.xml.

-Dmapreduce.job.counters.limit=x
-Dmapreduce.job.counters.groups.max=y

NOTE: x and y are custom values based on your environment/requirement.

edited Oct 13 '15 at 14:22

YoungHobbit

13,254
9
50
73

answered Oct 13 '15 at 14:09

Vijayanand

470
4
10

More than 120 counters in hadoop

5 Answers5

Linked