12

Is there a unified theory of caching? That is, collections of theorems and algorithms for constructing caches and/or optimizing for them?

The question is deliberately broad, because the results I am looking for are also broad. Formulas for maximum achievable speedup, metrics for caching algorithms, stuff like that. A university-level textbook would probably be ideal.

Jørgen Fogh
  • 7,516
  • 2
  • 36
  • 46

3 Answers3

3

The vast majority of real-world caching involves exploiting the "80-20 rule" or Pareto distribution. See below for how it looks

Pareto distribution

This manifests itself in applications as:

  • Much of the runtime is spent in the same piece of code (making code caches on CPUs effective)
  • Often, when a variable is accessed, it will be accessed again soon (making data caches on CPUs effective)
  • When a browser looks up a website's hostname once, it will access it quite frequently in the near future (making DNS caches effective)

Therefore, I would say the "theory of caching" is to use up just a few extra resources which are generally "rare" but "fast" to compensate for the most active repeated things you're going to do.

The reason you do this is to try to "level out" the number of times you do the "slow" operation based on the highly skewed chart above.

Community
  • 1
  • 1
Chris Harris
  • 4,705
  • 3
  • 24
  • 22
3

I talked to one of the professors at my school, who pointed me towards online algorithms, which seems to be the topic I am looking for.

There is a great deal of overlap between caching algorithms and page replacement algorithms. I will probably edit the WikiPedia pages for these topics to clarify the connection, once I have learned more about the subject.

Jørgen Fogh
  • 7,516
  • 2
  • 36
  • 46
2

If you can assume that a cache hit is much faster than a cache miss, you will find that overtime, even if you have only cache misses, using a cache will still be as fast or faster than not using a cache.

See below for the math:

Number of hits = NumRequests - #CacheMisses

AverageTime = ((NumRequests-#CacheMisses) * TimePerHit + #CacheMisses * TimePerMiss)/NumRequests

If we then assume that NumRequests is infinity (this is a limit problem, don't fear the calculus), we can see this:

AverageTime = Infinity*TimePerHit/Infinity - #CacheMisses*TimePerHit/Infinity + #CacheMisses*TimePerMiss/Infinity

Both terms with the #CacheMisses goes to zero, but the whole equation resolves to:

AverageTime = TimePerHit

Granted this is for when the number of requests is infinity, but you can see how this will easily speed up your system by using a cache.

samoz
  • 56,849
  • 55
  • 141
  • 195
  • Your calculation looks very good. Unfortunately it implies the assumption that the number of cache misses is constant, which seems highly unlikely. The correct number would be: HitProbability * TimePerHit + (1 - HitProbability) * TimePerMiss – Jørgen Fogh Jun 17 '09 at 12:04
  • I know the math is a little iffy; this is a proof I had to learn for a class I took last fall and I just tried to recall it from there. Addressing your point though, since I have CacheHits vs CacheMisses, this is a ratio, so wouldnt this be the same thing as using a hit probability? – samoz Jun 17 '09 at 12:48
  • Not quite. Assuming that there is a constant non-zero probability of cache misses, there will be infinitely many misses if infinite trials are performed. Your formula would be correct if the cache is big enough to hold every piece of data which will ever be requested. Then there is a constant upper bound to the number of misses. – Jørgen Fogh Jun 17 '09 at 13:48
  • Yeah, agree with Fogh. It also doesn't sound right at the first place. – Khue Vu Oct 28 '11 at 03:49
  • I'm basically saying that using a cache imposes a negligible overhead into a system. As the number of requests increases towards infinity, the overhead becomes essentially zero. – samoz Oct 29 '11 at 01:38