0

Imagine a websystem that's going to cache webpages; it's going to store about 10 million of 'm. The only thing I need to do is to return a webpage now and then from the cache, and update/delete webpages in the cache.

The approximate size of the cache will be 250GB, I do not need to search in the cache's contents. I expect about 70.000 requests/updates/deletes to the cache on daily basis.

The question: what suits best, a database or a file cache (2-sublevel directories). Do you recommend putting the data on a separate server or NAS?

Any help (and suggestions!) are really appreciated!

Adriaan

user59810
  • 109
  • 5

1 Answers1

2

In order to maintain a cache you need to be able to locate items based on the URL and the age (latter for clearing out LRU files). Using raw files you therefore would have to maintain an additional index or more than one link to the actual datafile. You've also got the comlpication that you need to distribute files evenly across directories to maintain efficiency in retrieving the content - e.g. you might use the md5 hash of the URL to generate the path to the content file / metadata.

So although database access would be slower, it does simplify the management of the metadata. That's not to say that the actual content could still reside in files - storing large, variable length blobs in a database is often less efficient than as ordinary files.

The other thing you need to think about is how to handle updates - how this is done depends a lot on how you implement the proxy server - as a STDIO server running from inetd, a forking or threaded server or a single threaded server. Except for the single-threaded server, there is a possiblity that 2 requests for the same content not available from the local cache may be processing at the same time - but you only want the proxy to refer to the origin once. Using a DBMS at the backend allows you a lot more flexibility over lock handling / concurrent access.

Do you recommend putting the data on a separate server or NAS

eh? You mean keep the storage on a seperate system from the proxy? Why on earth would you do that? Your just adding additional delays to accessing the content. And accessing database files over a network is never a good idea.

However this all re-supposes that you intend writing your own software. If you want to do it just for fun (?) then part of the process of learning to program is investigating these problems yourself - and testing different approaches - asking someone else how they would write it does not help you learn about the issues. OTOH if you just need a solution to the problem why not use one of the open-source packages available - squid is very good.

symcbean
  • 21,009
  • 1
  • 31
  • 52
  • Thanks for your reply. I'm using Zend Framework, which has cache adapters for file- and database caching (also taking care of metadata). So if i understand correctly, file caching is more efficient and therefore the way to go? I was thinking about accessing the files over the network because of sharing the server load. – user59810 Dec 01 '10 at 13:34