Custom Counter inside the Hadoop Partitioner

Question

I would like to capture some information on keys and their values inside a custom Partitioner (or even the default HashPartitioner).

I can use custom counters inside both mappers and reducers by accessing the "context" variable. However, inside the Partitioner there is no access to the "context" variable.

Is there any way to: -1- get access to the "context" variable from the Partitioner ? or -2- how to add a counter to the Partitioner ?

Thank you.

Chris Gerken · Answer 1 · 2013-04-16T20:08:45.183

1

Every key/value pair going through a partitioner either (1) was written by a mapper to a context or (2) will be passed to a reducer's reduce() method. You can put code in either of those two places to write and increment your custom counters to the context.

If you have to write code in the partitioner, then try making your partitioner implement JobConfigurable. That should make hadoop call your partitioner with a JobConf object when the partitioner is first instantiated. You can put information in your configuration (like the name of a sequential or text file to output information to) and pass it to your partitioner this way.

edited Apr 16 '13 at 20:08

answered Apr 16 '13 at 19:38

Chris Gerken

16,221
6
44
59

Thanks Chris for the answer. As you said, it is possible to add counters on both the mapper or the reducer. But what I really need, is an interceptor of those values without having access to the mapper or reducer code. Is there any way, to write interceptors in Hadoop ? – user2262938 Apr 16 '13 at 19:46

score 0 · Answer 2 · answered Apr 16 '13 at 23:56

There is no way to get access to the context (and as a result the counters) in the partitioner.

Not sure why you can't add your logic in the mapper as Chris Gerken suggests, but a similar / alternative approach would be to create a map wrapper class, where you create a Mapper.Context wrapping class and then delegate to the actual map method of your desired Mapper class with this wrapped context - then you can intercept the context.write method calls and do what you wish - not really any different from just updating the mapper code though.

Custom Counter inside the Hadoop Partitioner

2 Answers2