how to handle data skew in reducer?

Asked Feb 05 '16 at 17:59

Active Sep 21 '17 at 07:42

Viewed 871 times

I have a simple mapreduce job , where in for some of keys , number of values are millions in number . As a result of which reducer is not able to finish . I have gone through this link Hadoop handling data skew in reducer but not able to follow if there is any best practice available for such kind of scenarios. Can anyone please suggest the best way to handle such cases in mapreduce job ?

edited Sep 22 '17 at 17:48

Community

asked Feb 05 '16 at 17:59

KBR

Is your operation associative? if so you can use combiner. – kalyan chakravarthy Mar 28 '18 at 02:34

how to handle data skew in reducer?

0 Answers0