1

I've been reading into different computer science topics and thus far, the most difficult part for me has been understanding the concept of efficient algorithms. I guess my question is, in theory, why does efficiency of algorithm really matter when the software will be running on different hardware of different speeds?

I hope I have been very clear in describing the issue at hand.

Abulurd
  • 1,018
  • 4
  • 15
  • 31

4 Answers4

2

Hardware speed is important, but hardware differences typically result in a small change in speed (usually a constant factor). Constant factors are typically ignored in measurements of algorithmic efficiency (for algorithm analysis, we mostly use Big-O/Theta/Omega notation, which doesn't account for constant differences in run-time).

If you have a bad algorithm, it might not realistically finish no matter how fast your hardware is. Consider the difference between a Theta(n^2) algorithm and an Theta(n!) algorithm. If n=1000000, then you'll be hard pressed to find a computer that can perform Theta(n!) operations in a reasonable amount of time. However, you can definitely find a computer than can perform Theta(n^2) operations in a reasonable amount of time.

John Kurlak
  • 6,594
  • 7
  • 43
  • 59
  • 1
    Ehh. n is O(n!) and n^2 is O(n^2). Upper bounds aren't the be-all and end-all. In fact, they often wind up telling you surprisingly little about what's going to happen. – tmyklebu Sep 24 '14 at 02:54
  • 2
    @tmyklebu Sure... but often when computer scientists write O(x) they mean Theta(x) – John Kurlak Sep 24 '14 at 03:40
2

If you ask an academic or recently-educated programmer, they will say what matters is big-O, because an O(n log n) algorithm will always beat an O(n*n) algorithm, provided n is big enough.

But that's the point - n may not actually be that big.

Plus they tend to ignore constant factors.
I once heard Jon Bentley gently chiding academics that they really wouldn't mind to have their salary multiplied by 50, would they?

In the real world, constant factors are very important.

P.S. Here's an example of a 730x speedup, achieved over a series of six edits.

Community
  • 1
  • 1
Mike Dunlavey
  • 40,059
  • 14
  • 91
  • 135
  • 1
    My comment about that one is, "In the real world, `log(n)` is a smallish constant. For some companies it is a slightly larger constant." – btilly Sep 24 '14 at 05:21
1

Faster algorithms are faster than slower algorithms regardless of the speed of the underlying hardware.

If the CPU is slow, it becomes even more important to choose efficient algorithms so you can do less work.

SLaks
  • 868,454
  • 176
  • 1,908
  • 1,964
1

A common measure of algorithm efficiency, the Big-O notation, lets you compare rates of growth in time the algorithms take relative to each other, assuming that they run on the same hardware, and ignoring constant factors.

When hardware speeds go up, all algorithms speed up by roughly the same constant: if the speed of hardware goes up by the factor of three, all algorithms would be three times faster*. This means that the algorithms that were faster on the old hardware would still be faster on the new hardware.

Moreover, the influence of hardware speedup is a constant independent of the problem size. Depending on the way the algorithm scales with the size of the data, the speedup from improved hardware would be different for different algorithms.

For example, consider two algorithms, X that grows as O(n) and Y that grows as O(n2). Let's say you measure the time that it takes them to process a fixed amount of data. Speeding up the CPU by a factor of four would let X process roughly four times the amount of data in the same time, while Y would be able to process only twice as much data (also approximately).

At the same time, hardware optimizations could give disproportionally large speed-ups to some operations, which, if useful to algorithm X and not useful to algorithm Y, would distort the relative speeds of the two algorithms running on different hardwares.

* Unless your algorithm hit a different bottleneck: for example, it is possible that the CPU speed-up of three times would need a matching speed-up in memory access in order to achieve the expected speed-up across all algorithms.

Sergey Kalinichenko
  • 714,442
  • 84
  • 1,110
  • 1,523
  • Your second paragraph hasn't been true for quite some time. We have more granular notions of "speed of a computer" because we keep coming up with increasingly clever computer architectures. Increasingly clever computer architectures mean that there are more different bottlenecks that can be hit. – tmyklebu Sep 24 '14 at 02:52
  • And your first paragraph isn't really true; O-notation lets you compare **rates of growth** of running times ignoring constant factors. It winds up saying nothing at all about anything particular; all you can say with it is that *eventually* something will happen---with no idea of when. – tmyklebu Sep 24 '14 at 02:53
  • @tmyklebu can you please back-up your claims with some evidence; I would like to look into this more. – Abulurd Sep 24 '14 at 02:59
  • Now that you've added a couple more paragraphs: Your fourth paragraph is also false. First, "Y would only process twice as much" assumes the running time is exactly C*n^2, not merely O(n^2). Further, something other than shoving instructions through the CPU may well become a bottleneck when you speed up the CPU by a factor of four or already be a bottleneck beforehand. – tmyklebu Sep 24 '14 at 03:01
  • @Wehelie: Go read the wikipedia page on O-notation. Then go compare the speed adding up the elements of an array in sequential order versus random order on a modern computer versus, say, an 80386. – tmyklebu Sep 24 '14 at 03:07
  • 1
    @tmyklebu I understand all that. I presented a rough approximation that should be sufficient to someone who wants to know why it is important to learn about algorithm efficiency, not to someone preparing to take his Ph.D. qualifiers in algorithm analysis. – Sergey Kalinichenko Sep 24 '14 at 03:09
  • 2
    @dasblinkenlight Hardware improvements DO NOT always affect all algorithms equally. For instance the current migration from traditional disks to high performance flash drives makes hash based algorithms worth considering in cases where sort based algorithms were the only choice worth thinking about. – btilly Sep 24 '14 at 05:20
  • @dasblinkenlight: Yeah, I believe you. The trouble with layman-oriented descriptions of O-notation is that they extrapolate a result "at infinity" to try to say something everywhere, and yours is no exception. A bound expressed in O-notation is as good as no bound at all if you're trying to say something concrete about something concrete. Higher order does not mean lower error in numerical analysis, and lower order of growth does not mean faster here. – tmyklebu Sep 24 '14 at 05:35
  • @btilly "Hardware improvements DO NOT always affect all algorithms equally." My last paragraph before the footnote says pretty much the same thing. – Sergey Kalinichenko Sep 24 '14 at 09:54