11

Somewhat of an odd question, but does anyone know what kind of sort MapReduce uses in the sort portion of shuffle/sort? I would think merge or insertion (in keeping with the whole MapReduce paradigm), but I'm not sure.

SubSevn
  • 1,008
  • 2
  • 10
  • 27

1 Answers1

11

It's Quicksort, afterwards the sorted intermediate outputs get merged together. Quicksort checks the recursion depth and gives up when it is too deep. If this is the case, Heapsort is used.

Have a look at the Quicksort class:

org.apache.hadoop.util.QuickSort

You can change the algorithm used via the map.sort.class value in the hadoop-default.xml.

Alexandre Dulaunoy
  • 130
  • 3
Thomas Jungblut
  • 20,854
  • 6
  • 68
  • 91
  • 1
    This switch of sort technique is a standard hybrid sorting AFAIK. Its called IntroSort: http://en.wikipedia.org/wiki/Introsort – Yash Sharma Feb 20 '14 at 18:44