Did I just prove that sieve of Eratosthenes is less efficient than trial division?

Question

I was trying to compare the run-time speed of two algorithms: A brute-force C program to print prime numbers (10,000 numbers), and a Sieve of Eratosthenes C program (also 10,000 prime numbers).

My measured run-time for the sieve algorithm was: 0.744 seconds

My measured run-time for the brute-force algorithm was: 0.262 seconds

However, I was told that the Sieve of Eratosthenes algorithm is more efficient than the brute-force method, and so I thought it would run faster. So either I'm wrong or my program is flawed (which I doubt).

Therefore, my question is: Since I got the opposite results than what I expected, does this prove that the Sieve of Eratosthenes really is the less efficient algorithm in terms of speed, compared to the trial division?

I'm not sure if it is of any relevance, but I'm using the Dev C++ compiler and Windows 7.

Short answer is: not necessarily. Efficiency is defined asymptotically (for arbitrarily large input). A lot of very efficient algorithms get their efficiency from some form of caching/preallocation... which make them slower than brute force for tiny inputs. In your case though, I find it VERY surprising that the brute force performs better than the sieve of eratosthenes (which is basically a 'clever' brute-force). Are you 100% sure of your implementation of it ? — val, Aug 16 '13 at 09:29
It depends on how algorithms are implemented. I solved that problem on project euler use sieve of eratosthenes, and it is much faster. — D.Pham, Aug 16 '13 at 09:32
Most benchmark tests in Windows are very inaccurate and often incorrect. Whenever discussing the measurement of execution time in Windows, you must mention how you measured the time. Also, since Windows 7 is not a RTOS, you cannot simply measure the time once, you'll have to do it several times and calculate the mean or average time. You also have to benchmark the two algorithms at the same time. If you measure algorithm A 100 times, and then suppose that some process starts in the background when you are about to measure algorithm B. Then you obviously get incorrect results. — Lundin, Aug 16 '13 at 09:48
And now its time for [Sieve of Atkin](http://en.wikipedia.org/wiki/Sieve_of_Atkin) — P0W, Aug 16 '13 at 09:51
The correct way to ask this question would have been to **post your code** as used to implement and time both of these algorithms. Then we wouldn't have to simply speculate as to what might be the cause of the behavior you're seeing under certain circumstances. We could *actually tell you what was wrong* with your code. The question has since been put "on hold" because it is incomplete. You can [edit] it to include your code and a more detailed description of your problem in order to get it re-opened. — Cody Gray - on strike, Aug 16 '13 at 10:23
"So either I'm wrong or my program is flawed (which I doubt)." -- I can't imagine why you would doubt it ... odds are your program doesn't even implement a Sieve of Eratosthenes; it's a common mistake even among experienced programmers: http://www.cs.hmc.edu/~oneill/papers/Sieve-JFP.pdf — Jim Balter, Aug 16 '13 at 13:00
as happens more and more often now, the given reason for closure is **[self-censored]**. It is perfectly reasonable to give a "few paragraphs" answer to this question. Here's the outline: ***measure* [empirical orders of growth](http://en.wikipedia.org/wiki/Analysis_of_algorithms#Empirical_orders_of_growth) *for both algorithms. One point isn't enough; three points are aplenty."*** That's all, and it's not among answers given here. — Will Ness, Aug 17 '13 at 11:38
[Meta discussion about this question](http://meta.stackexchange.com/questions/193501/why-does-this-question-appear-to-be-acceptable) — Gilles 'SO- stop being evil', Aug 18 '13 at 19:28
@CodyGray we can still discuss whether the one size-point measurement means anything or must the comparison be made for more sizes of input? Even if we know nothing about the code in question, there is an answer to that question. :) I've edited to make the question more specific. — Will Ness, Aug 19 '13 at 16:55
@Will Sure, you'll notice I wasn't one of the people who voted to close. I posted that comment as some constructive advice for the asker, after reading the Meta question that Gilles already linked to. Being a regular participant on Meta, I could see the writing on the wall: this question was about to be closed, and experience told me that the problem was likely related to the lack of code. There are lots of people who think [the guideline in the FAQ](http://stackoverflow.com/help/on-topic) is a bit more concrete: good SO questions need to have code. — Cody Gray - on strike, Aug 20 '13 at 07:34
For the record, I've removed the superfluous formatting that left an initial bad taste in my mouth when looking at this question, and cast a vote of my own to re-open. I still think it would be an improvement if the code used to test the problem were *also* posted, but that doesn't mean I think the question is unacceptable in its current state. — Cody Gray - on strike, Aug 20 '13 at 07:37
@CodyGray: A good attempt at fixing it, but I still think the code used for the algorithms needs to be included. The question states "does this prove that the Sieve of Eratosthenes really is the less efficient algorithm", for all we know the reason for the slower time could purely be down to a bad implementation. I think the only possible answer that could be given is "The Sieve of Eratosthenes algorithm, with your specific implementations, is definitely slower"... but that's kind of like asking "Which is faster, 2 second or 7 seconds?"... pointless — musefan, Aug 20 '13 at 08:10
to the new close voter: why don't you *post a comment* explaining what's still "unclear" here? (I can't imagine what that might be, please explain). Perhaps the question can be clarified without going through the motions of closing and reopening it yet again. — Will Ness, Aug 21 '13 at 13:46
The code is still missing after a day. Without the code this is completely worthless and should be closed and deleted. — starblue, Aug 21 '13 at 22:21
@starblue I obviously strongly disagree as can be seen from my answer. There is a meaningful discussion to be had about it, and a meaningful answer. Which I hope I gave. It is worthwhile - *important* - to know that *whatever* the code, its efficiency is best assessed with empirical orders of growth technique, and measuring comparative speed at just one point of input size is meaningless. — Will Ness, Aug 22 '13 at 06:01

Will Ness · Accepted Answer · 2014-06-14T13:33:13.380

TL;DR: comparing the speed of code variants at just one input size is meaningless; comparing empirical orders of growth truly reflects algorithmic nature of the code and will be consistent across different test platforms, for the same test range of input sizes. Comparing absolute speed values is only meaningful for code variants which exhibit same asymptotic or at least local growth behaviour.

It is not enough to measure speed of your two implementations just at one input size. Usually several data points are needed, to assess the run time empirical orders of growth of our code (because the code can be run with varying input sizes). It is found as the logarithm of the ratio of run times, in base of the ratio of input sizes.

So even if at some input code_1 runs 10 times faster than code_2, but its run time doubles with each doubling of the input size, whereas for code_2 it only grows as 1.1x, very soon code_2 will become much much faster than code_1.

So the real measure of an algorithm's efficiency is its run time complexity (and the complexity of its space i.e. memory requirements). And when we measure it empirically, we only measure if for the particular code at hand (at a particular range of input sizes), not for the algorithm itself, i.e. the ideal implementation of it.

In particular, the theoretical complexity of trial division is O(n^1.5 / (log n)^0.5), in n primes produced, usually seen as ~ n^1.40..1.45 empirical order of growth (but it can be ~n^1.3 initially, for smaller input sizes). For the sieve of Eratosthenes it is O(n log n log (log n)), seen usually as ~ n^1.1..1.2. But there certainly are sub-optimal implementations of both the trial division and the sieve of Eratosthenes that run at ~n^2.0 and worse.

So no, this proves nothing. One datapoint is meaningless, at least three are needed to get a "big picture" i.e. to be able to predict with some certainty the run time ⁄ space needed for bigger input sizes.

Prediction with known certainty is what the scientific method is all about.

BTW your run times are very long. The calculation of 10,000 primes should be nearly instantaneous, much less than 1/100th of a second for a C program run on a fast box. Perhaps you're measuring printing time as well. Don't. :)

Jacob Pollack · Answer 2 · 2013-08-16T10:10:00.067

No, elapsed run time is not a standard for measuring efficiency as it varies from platform to platform -- saying "my algorithm ran in 10 seconds" gives little to no information about the algorithm itself. In addition to that, you would need to list the entire environment specs and other processes running at the same time and it would be a huge mess. Hence, the development of the order notations (Big Oh, Little Oh, Omega, etc.).

Efficiency is typically branched into two subsections:

Time efficiency.
Space efficiency.

... where one algorithm may be extremely time efficiency, but very inefficient space-wise. Vice-versa applies. Algorithms are analyzed based on their asymptotic behaviour when scaling the amount of instructions they need to execute for a given input n. This is a very high-level explanation on a field that is meticulously studied by PhD Computer Scientists -- I suggest you read more about it here for the best low-level explanation that you will find.

Note, I am attaching the link for Big Oh notation -- the sister notations can all be found off of that Wikipedia page and it's typically a good place to start. It will get into the difference of space and time efficiency as well.

Small Application of Time Efficiency using Big Oh:

Consider the following recursive function in Racket (would be in Python if I knew it -- best pseudo code I can do):

(define (fn_a input_a)
  (cond
    [(empty? input_a) empty]
    [(empty? (rest input_a)) input_a]
    [(> (first input_a) (fn_a (rest input_a))) (cons (first input_a) empty)]
    [else (fn_a (rest input_a))]))

... we see that: empty?, rest, > and first are all O(1). We also notice that in the worst case, a call is made to fn_a in the third condition and fourth condition on the rest of input_a. We can then write our recurrence relation as, T(n) = O(1) + 2T(n - 1). Looking this up on a recurrence relation chart we see that fn_a is of order O(2^n) because in the worst case, two recursive calls are made.

It's also important to note that, by the formal definition of Big Oh it is also correct (however useless) to state that fn_a is O(3^n). Lot's of algorithms when analyzed are stated using Big Oh however it would be more appropriate to use Big Theta to tighten the bounds, essentially meaning: the lowest, most accurate order with respect to a given algorithm.

Be careful, read the formal definitions!

Runtime means something to my users. Certainly big-O has constant factors in so *that* way of analysing "efficiency" might not be directly proportional to runtimes. — doctorlove, Aug 16 '13 at 09:33
@doctorlove, improved the wording I used. It should be communicated better now (towards my original intention). — Jacob Pollack, Aug 16 '13 at 09:37
Better :-) I still suspect if the OP has run the two algos on the same machine the times *might* indicate a code screw up, but it would need more than one experiment — doctorlove, Aug 16 '13 at 09:39
I completely agree with Jacob, while runtime (and pay attention that it's also not the same as wall clock time in systems where you may be preempted), is important and the constant factors are to be reduced - the complexity notation is infinitely more important when scaling N, which is in many cases the bottom line — Leeor, Aug 16 '13 at 09:41
or you can just [measure it empirically](http://en.wikipedia.org/wiki/Analysis_of_algorithms#Empirical_orders_of_growth), to get the big picture. — Will Ness, Aug 19 '13 at 13:01

score 2 · Answer 3 · answered Aug 16 '13 at 09:38

Does a longer run-time mean a less efficient algorithm?

Not necessary. The efficiency of program is measured not only by the time it takes but also by the resources which it is taking. Space is one other factor which is kept in mind while considering the efficiency.

From the wiki:-

For maximum efficiency we wish to minimize resource usage. However, the various resources (e.g. time, space) can not be compared directly, so which of two algorithms is considered to be more efficient often depends on which measure of efficiency is being considered as the most important, e.g. is the requirement for high speed, or for minimum memory usage, or for some other measure?

score 1 · Answer 4 · answered Aug 16 '13 at 09:30

In general: yes, but when you are down in the sub 1 second range, there is a lot of noise which could be confusing...

Runs each test many many times and use some stats on the results (e.g. average or mean/deviation depending on how much you care)

And/or make it do more work - like finding a larger number of primes

score 1 · Answer 5 · answered Aug 16 '13 at 09:30

In short, yes, if by efficiency you mean time efficient. There are memory considerations too.

Be careful how you measure though - make sure your timing tools are precise.

Make sure you measure on the same machine when nothing else is running.
Make sure you measure several times and take an average and the varinace for a decent comparison.
Consider getting someone to review your code to check it is doing what you think it is doing.

score 1 · Answer 6 · answered Aug 16 '13 at 09:35

The efficiency of algorithms is normally measured by how efficiently they handle large inputs. 10,000 numbers is not a very large input, so you may need to use a larger one before the sieve of Eratosthenes starts to become quicker.

Alternatively, there may be a big in one of your implementations

Lastly, efficiency in algorithms can be measured by amount of memory needed (but this measure is less common, especially since memory is so cheap nowadays)

Did I just prove that sieve of Eratosthenes is less efficient than trial division?

6 Answers6

Linked