34

Isn't Insertion sort O(n^2) > Quicksort O(n log n)...so for a small n, won't the relation be the same?

mitchus
  • 4,677
  • 3
  • 35
  • 70
user1031752
  • 395
  • 2
  • 6
  • 9

6 Answers6

29

Big-O Notation describes the limiting behavior when n is large, also known as asymptotic behavior. This is an approximation. (See http://en.wikipedia.org/wiki/Big_O_notation)

Insertion sort is faster for small n because Quick Sort has extra overhead from the recursive function calls. Insertion sort is also more stable than Quick sort and requires less memory.

This question describes some further benefits of insertion sort. ( Is there ever a good reason to use Insertion Sort? )

Community
  • 1
  • 1
Casey Robinson
  • 1,021
  • 9
  • 17
14

Define "small".

When benchmarking sorting algorithms, I found out that switching from quicksort to insertion sort - despite what everybody was saying - actually hurts performance (recursive quicksort in C) for arrays larger than 4 elements. And those arrays can be sorted with a size-dependent optimal sorting algorithm.

That being said, always keep in mind that O(n...) only is the number of comparisons (in this specific case), not the speed of the algorithm. The speed depends on the implementation, e. g., if your quicksort function as or not recursive and how quickly function calls are dealt with.

Last but not least, big oh notation is only an upper bound.

If algorithm A requires 10000 n log n comparions and algorithm B requires 10 n ^ 2, the first is O(n log n) and the second is O(n ^ 2). Nevertheless, the second will (probably) be faster.

Dennis
  • 14,264
  • 2
  • 48
  • 57
  • 4
    For the curious, the `O(N^2)` one will be faster than the `O(N Log N)` one until about `N=9000` entries or so. – sarnold Nov 12 '11 at 01:04
  • 4
    Big Oh notation is not an upper bound. It characterizes the asymptotic behavior of a function. – Casey Robinson Nov 12 '11 at 01:10
  • @sarnold: That would be `20,000,000` vs `100,000` comparisons. No way. – Dennis Nov 12 '11 at 01:35
  • @Dennis: [Check Wolfram Alpha](http://www.wolframalpha.com/input/?i=10000+*+x+*+log(x)+%3D%3D+10+*+x^2). Or `echo "10000 * 9000 * l(9000) ; 10 * 9000 * 9000" | bc -l`. – sarnold Nov 12 '11 at 01:39
  • 1
    @CaseyRobinson: No. It does not characterize the asymptotic behavior, it just describes it. For example, any `O(n ^ 2)` algorithm is automatically also `O(n ^ 3)`. And `f(n) = O(n ^ 2)` means that there is some `k` such that `|f(n)| <= k n ^ 2`. That is an upper board. – Dennis Nov 12 '11 at 01:40
  • 1
    @sarnold: My bad. I thought you were talking about quicksort and insertion sort. – Dennis Nov 12 '11 at 01:41
  • @Dennis From Wikipedia "Big O notation characterizes functions according to their growth rates" http://en.wikipedia.org/wiki/Big_O_notation – Casey Robinson Nov 12 '11 at 01:43
  • @Dennis: No wonder! :) One more reminder that even in comments it helps to be precise... – sarnold Nov 12 '11 at 01:43
  • 1
    @CaseyRobinson: From the same paragraph of the same article: *A description of a function in terms of big O notation usually only provides an upper bound on the growth rate of the function.* – Dennis Nov 12 '11 at 01:44
  • @Dennis We are saying the same thing and arguing about semantics. – Casey Robinson Nov 12 '11 at 01:51
5

O()-notation is typically used to characterize performance for large problems, while deliberately ignoring constant factors and additive offsets to performance.

This is important because constant factors and overhead can vary greatly between processors and between implementations: the performance you get for a single-threaded Basic program on a 6502 machine will be very different from the same algorithm implemented as a C program running on an Intel i7-class processor. Note that implementation optimization is also a factor: attention to detail can often get you a major performance boost, even if all other factors are the same!

However, the constant factor and overhead are still important. If your application ensures that N never gets very large, the asymptotic behavior of O(N^2) vs. O(N log N) doesn't come into play.

Insertion sort is simple and, for small lists, it is generally faster than a comparably implemented quicksort or mergesort. That is why a practical sort implementation will generally fall back on something like insertion sort for the "base case", instead of recursing all the way down to single elements.

comingstorm
  • 25,557
  • 3
  • 43
  • 67
3

Its a matter of the constants that are attached to the running time that we ignore in the big-oh notation(because we are concerned with order of growth). For insertion sort, the running time is O(n^2) i.e. T(n)<=c(n^2) whereas for Quicksort it is T(n)<=k(nlgn). As c is quite small, for small n, the running time of insertion sort is less then that of Quicksort.....

Hope it helps...

bhuwansahni
  • 1,834
  • 1
  • 14
  • 20
2

Good real-world example when insertion sort can be used in conjunction with quicksort is the implementation of qsort function from glibc.

The first thing to point is qsort implements quicksort algorithm with a stack because it consumes less memory, stack implemented through macros directives.

Summary of current implementation from the source code (you'll find a lot of useful information through comments if you take a look at it):

  1. Non-recursive

  2. Chose the pivot element using a median-of-three decision tree

  3. Only quicksorts TOTAL_ELEMS / MAX_THRESH partitions, leaving insertion sort to order the MAX_THRESH items within each partition. This is a big win, since insertion sort is faster for small, mostly sorted array segments.

  4. The larger of the two sub-partitions is always pushed onto the stack first

What is MAX_THRESH value stands for? Well, just a small constant magic value which

was chosen to work best on a Sun 4/260.

funnydman
  • 9,083
  • 4
  • 40
  • 55
0

How about binary insertion sort? You can absolutely search the position to swap by using binary search.

phatnhse
  • 3,870
  • 2
  • 20
  • 29
  • `position to swap` is a misunderstanding: *position to insert*, and you have to make space for that. Most of the time, "array" is implied: you have to move/copy every item to one side of "*the* position". Probably, you have read keys of items "to the other side of *the* position" during binary search that would not have been touched using linear search: detrimental with key access on critical path. – greybeard Mar 20 '18 at 10:36
  • Agree. That's the trade-off. You have to choose in this situation. Using binary search decrease the steps and quickly give you the position to insert, but you need more space to do the "array-copy". Worst case still linear time, but in average, I think it more effecient than O(n) – phatnhse Mar 20 '18 at 11:16
  • `I think [binary finding the insertion point] more [efficient] than O(n)` - dominated by `the "array-copy"`, lower constant for bulk copy or not. – greybeard Mar 20 '18 at 11:21