Hadoop configuration - are mapper/combiner affected by io.sort.factor and io.sort.mb?

Question

If I modify the io.sort.factor and io.sort.mb, does the local sort taking place on the map side use these variables, or are they only used by the sort done on the reducer side?

score 2 · Accepted Answer · answered Jun 20 '12 at 00:03

2

Yes, they are used on the map side too (irrespective of whether you have a combiner or not):

MapTask.java
- io.sort.factor - line 1695
- io.sort.mb - lines 932 - 944

answered Jun 20 '12 at 00:03

Chris White

29,949
4
71
93

Hadoop configuration - are mapper/combiner affected by io.sort.factor and io.sort.mb?

1 Answers1