9

I am using Varnish 3.0.3 (the latest as of this writing) with more or less the default install, using the following for storage:

# # Cache file location
VARNISH_STORAGE_FILE=/var/lib/varnish/varnish_storage.bin
#
# # Cache file size: in bytes, optionally using k / M / G / T suffix,
# # or in percentage of available disk space using the % suffix.
VARNISH_STORAGE_SIZE=4000M
#
# # Backend storage specification
VARNISH_STORAGE="file,${VARNISH_STORAGE_FILE},${VARNISH_STORAGE_SIZE}"
#

Most of the pages that are can be cached will have a very long TTL. The site I'm working with is huge, and has millions of pages, which can't possibly fit into the storage size listed above (4 GB).

Set up for the questions:

  • Let's assume the entire storage space is full, and nothing in the cache is expired yet (my TTL is 2 weeks on every single item in the cache).

  • I'm assuming the Ban Lurker is useless here, because it will only delete items that are expired, and none are expired.

Questions:

  • Given the above scenario, what happens to Varnish when additional requests come in for something that's not already in the cache? Does Varnish make any attempt to delete older items from the cache, regardless of the TTL of those items? Does Varnish crash? Hang?

  • If Varnish doesn't handle the scenario, is there something I can do to configure it to handle the scenario better? (I could obviously lower the TTL of the items in the cache, but I'm wondering if there are any additional suggestions).

I'm seeing in this scenario that effectively Varnish stops responding to requests. At the point when this happens, I'm not out of memory or CPU on the box.

Thanks.

Follow-up:

I also saw this post

which pointed me towards

which seems to indicate everything "just happens"... but it looks like it actually sends a PURGE request for the URL, which may require adding the appropriate stuff to your VCL(see https://www.varnish-cache.org/docs/3.0/tutorial/purging.html)

I am going to try that and will follow-up if that solves the issue.

Community
  • 1
user174548
  • 91
  • 1
  • 3
  • 3
    As far as we are concerned, we use Varnish for caching huge amounts of data too, specifically, an image repository of about 2,8 million images with like a 30 gig Varnish file. Varnish fill fill up the 30 gig and as soon as new things get added to the cache, deletes the cache entry that was "not accessed for the longest time". Every cache entry has a time associated with it, it's the last access timestamp. The entry with the lowest last access timestamp goes. (Btw: Why do we cache images!? Because the HDD is too slow to serve them and SSDs for everything costs too much) – sinni800 May 23 '13 at 06:16

1 Answers1

1

According to your setup, you seem to use disk for storing varnish cache.

This is not recommended (for performance) unless you have a good reason (like sinni800).

Try using RAM for storing varnish cache ("malloc" instead of "file") and reproduce the issue.