1

I have a Hadoop job and define several counters by myself. When I am running this job (numbers of map > 500), I found some map tasks lost the user defined counters, but the Hadoop built-in counters (like map input records) work fine. The problem occours randomly on some DataNode. It is confusing. So what is the problem?

Usama Abdulrehman
  • 1,041
  • 3
  • 11
  • 21
user1573269
  • 363
  • 1
  • 4
  • 9
  • Can you post any code having to do with the counters you've defined? – Chris Gerken Nov 27 '12 at 03:37
  • I feel what you are facing is not an issue. We cannot expect to see the custom counter on all the map tasks.Only the map tasks which actually executed the code that increments your custom counter will show it.In other words, if the data processed by a map task do not fall into the criteria you are using to increment the counter, the custom counter wont show up in the counters list for that map task. – Eswara Reddy Adapa Nov 27 '12 at 07:19
  • @Eswara No, I just increase the counter when process a row. So the processed counter is equal to map input records. But by monitoring the task counters, I found some tasks have defined task, some do not. But every task has the map input records task. – user1573269 Nov 27 '12 at 09:31

1 Answers1

0

Wrap your map() method logic with a try-catch block and in the catch block increment another counter named after the exception message:

@Override
public void map(ByteBuffer key,SortedMap<ByteBuffer, IColumn> value, Context context)
        throws IOException, InterruptedException {

    try {

        //  Map logic goes here.

        context.getCounter("Custom","Counter").increment(1);

    } catch (Exception e) {
        context.getCounter("Exception",e.getMessage()).increment(1);
    }
    // End map logic 

}

It's possible that your row processing throws exceptions and thus bypasses the increment logic.

Chris Gerken
  • 16,221
  • 6
  • 44
  • 59