4

Comparison based sorting is big omega of nlog(n), so we know that mergesort can't be O(n). Nevertheless, I can't find the problem with the following proof:

Proposition P(n): For a list of length n, mergesort takes O(n) time.

P(0): merge sort on the empty list just returns the empty list.

Strong induction: Assume P(1), ..., P(n-1) and try to prove P(n). We know that at each step in a recursive mergesort, two approximately "half-lists" are mergesorted and then "zipped up". The mergesorting of each half list takes, by induction, O(n/2) time. The zipping up takes O(n) time. So the algorithm has a recurrence relation of M(n) = 2M(n/2) + O(n) which is 2O(n/2) + O(n) which is O(n).

rjkaplan
  • 3,138
  • 5
  • 27
  • 33
  • Why does the sorting of each half-list take O(n/2)? – Lasse V. Karlsen Nov 14 '11 at 20:29
  • By induction, we assumed P(n) to be true for 1 through n-1. That is: we assumed that for lists of length 1 through n-1, mergesort takes linear time on them. Specifically, for a list of length n/2 it takes O(n/2) time. – rjkaplan Nov 14 '11 at 20:32

3 Answers3

7

Compare the "proof" that linear search is O(1).

  1. Linear search on an empty array is O(1).
  2. Linear search on a nonempty array compares the first element (O(1)) and then searches the rest of the array (O(1)). O(1) + O(1) = O(1).

The problem here is that, for the induction to work, there must be one big-O constant that works both for the hypothesis and the conclusion. That's impossible here and impossible for your proof.

Per
  • 2,594
  • 12
  • 18
  • Hmm. What about the proposition *P(n): T(n) < c n* for some c. Then the last step becomes *T(n) = 2T(n/2) + n < 2c (n/2 )+ n* by induction. But this is just *(c + 1) n*. That is: the constant we're looking for is *(c + 1)*. We can choose c = 1. Then, I think, it works for the base case and the recursion. – rjkaplan Nov 14 '11 at 20:49
  • 3
    Ah, but if you choose _c(k)_ minimal so that for all _m_ < _k_, you have _T(m) <= c*m_, then the step shows that you can't deduce _c(n) = c(n/2)_, but only _c(n) <= c(n/2) + 1_, that means you can only deduce _c_ is O(log _n_). – Daniel Fischer Nov 14 '11 at 21:06
  • @rjkaplan To put it another way, in order for the math to work when the big-O is gone, you need *there exists **c** such that for all **n** ...*, but what you get with that argument is *for all **n**, there exists **c**...*. – Per Nov 14 '11 at 21:42
  • @DanielFischer. What do you mean by c(k)? What is k? – rjkaplan Nov 14 '11 at 22:08
  • @rjkaplan `c(k) = sup {T(m)/m : 1 <= m <= k}`. `k` any positive integer. Saying that `T` is in `O(n)` then is equivalent to saying that `c(k)` is bounded (which it isn't). – Daniel Fischer Nov 14 '11 at 22:38
4

The "proof" only covers a single pass, it doesn't cover the log n number of passes.

The recurrence only shows the cost of a pass as compared to the cost of the previous pass. To be correct, the recurrence relation should have the cumulative cost rather than the incremental cost.

You can see where the proof falls down by viewing the sample merge sort at http://en.wikipedia.org/wiki/Merge_sort

Raymond Hettinger
  • 216,523
  • 63
  • 388
  • 485
  • What is wrong is the line "The mergesorting of each half list takes, by induction, O(n/2) time". It should say, "the merge sorting of each half list takes O(n/2) time *plus* the time to merge sort its quarter lists." – Raymond Hettinger Nov 14 '11 at 20:38
  • I see. But the recurrence relation on that page is the same as mine. Copied and pasted, it is: T(n) = 2T(n/2) + n. I thought that the point of recurrences what that you're abstracting out the cumulative cost. – rjkaplan Nov 14 '11 at 20:40
  • Ah, nope. I disagree. "mergesorting each half list" implies that you are sorting **and zipping up** its half-lists. Those two operations are what comprises mergesort. That is: we have the recurrence relation T(n) = 2T(n/2) + n where T(n/2) = 2T(n/4) + ***n/2***. That last n/2 accounts for zipping up the quarter lists. – rjkaplan Nov 14 '11 at 20:43
  • Consider a recurrence relation for the Fibonacci series, ``F(n) = F(n-1)+F(n-2)``. That produces the next element in the series but not the cumulative sum of the F(0)..F(n). In your case, you want the cumulative cost of the mergesort, not just the cost of the immediately previous step. – Raymond Hettinger Nov 14 '11 at 21:42
  • Sure. T(n) = 2 T(n/2) + n does this. Because T(n/2) = 2T(n/4) + n/2, and T(n/4) = 2T(n/8) + n/4, and T(n/8) = 2T(n/16) + n/8, etc... – rjkaplan Nov 14 '11 at 21:59
  • 1
    `T(n) = n + 2T(n/2) = n + 2(n/2 + 2T(n/4)) = 2n + 4T(n/4) = 2n + 4(n/4 + 2T(n/8)) = 3n + 8T(n/8) = ... = kn + 2^k*T(n/2^k)` which, for `2^k ~ n` means `T(n) = n*log n`. – Daniel Fischer Nov 14 '11 at 23:21
  • @DanielFischer Nice mathematical summary :-) – Raymond Hettinger Nov 14 '11 at 23:30
3

Here is the crux: all induction steps which refer to particular values of n must refer to a particular function T(n), not to O() notation!

O(M(n)) notation is a statement about the behavior of the whole function from problem size to performance guarantee (asymptotically, as n increases without limit). The goal of your induction is to determine a performance bound T(n), which can then be simplified (by dropping constant and lower-order factors) to O(M(n)).

In particular, one problem with your proof is that you can't get from your statement purely about O() back to a statement about T(n) for a given n. O() notation allows you to ignore a constant factor for an entire function; it doesn't allow you to ignore a constant factor over and over again while constructing the same function recursively...

You can still use O() notation to simplify your proof, by demonstrating:

T(n) = F(n) + O(something less significant than F(n))

and propagating this predicate in the usual inductive way. But you need to preserve the constant factor of F(): this constant factor has direct bearing on the solution of your divide-and-conquer recurrence!

comingstorm
  • 25,557
  • 3
  • 43
  • 67