11

I have two servers behind a load balancer. Each server is running a memcached server and the settings file (which is identical on both servers) has them both defined (in short: shared cache).

I want the paths to the generated files to be identical on the servers so that the client does not have to download more than once.

For me to get this working I need to understand how django compressor works.

  • What is the actual purpose of the cache in django compressor?
  • Is the file content stored in both cache and on filesystem?
    • If so, which happens first?
  • I hope I'm asking the right questions here. Feel free to add some.

A more detailed and better constructed sequence than this would be very helpful.

Edit

  • Since the servers both share a memcached server, should I set COMPRESS_CACHE_KEY_FUNCTION = 'compressor.cache.socket_cachekey' (see develop branch) or does using the same cache key contribute to my point of having the same file names?
  • The way I understand this, mtime is collected from the source js/css files to determine if they may have changed and a new file should be generated out of them. Correct?
    • This probably does not happen on each and every load. When does it happen?
demux
  • 4,544
  • 2
  • 32
  • 56
  • 1
    If i was you and want to know such details about django-compressor, I would read the code (django-compressor code). – Tomasz Wysocki Aug 30 '11 at 18:43
  • 1
    I've already done that. But although I can understand what most of the code is doing part for part, I fail to comprehend the bigger picture of it if you know what I mean. So I thought: Maybe someone has been working on django-compressor more than I have and can explain to me how it works so that I can better comprehend what to make of the code while looking at it. – demux Aug 31 '11 at 01:05

3 Answers3

12

In the develop branch there is a new option to change the css hashing method. https://github.com/jezdez/django_compressor

See line 61 in filters/css_default.py

The settings I'm using:

COMPRESS_ENABLED = True
COMPRESS_OFFLINE = False
COMPRESS_STORAGE = 'compressor.storage.GzipCompressorFileStorage'
COMPRESS_CSS_HASHING_METHOD = 'hash' # not using mtime since it differs between servers.

There is no option like this for js files since their hash key is never generated using mtime anyway.

This works perfectly behind my loadbalancer.

When this is written, the following is the latest commit in the develop branch: https://github.com/jezdez/django_compressor/commit/d48bc5f45d5a55b0f826eb605ccf09a6bf33fcb9

demux
  • 4,544
  • 2
  • 32
  • 56
4

If you want to have identical cache files you must be sure that you have identical input on both servers.

You should check:

  • if code in {% compress %}...{% endcompress %} is identical on both servers (if you deploying to both servers at once it should be)
  • if all your .css/.js files are identical on both servers (if you deploying to both servers at once it should be)
  • if mtime (modify time) of your .css/.js files is identical on both servers (your deployment script may affect those and set current date)

If all of this requirements ale satisfied, generated files should be identical (content and names).

You can check mtime using "stat" unix command.

Answers to your questions:

  • Purpose of the cache in django-compressor is to reduce reads from file system.
  • Generated file with combined code is stored only on filesystem.

Edit:

I have checked it on one of my websites behind load balancer. I have different file names for .css files, but they are identical for .js.

For .css files I use preprocessor (http://lesscss.org/), so it affects mtime.

Edit (after topic developed):

What is in the cache?

Due to documentation django-compressor stores in the cache two different things:

  • mtime of cached files (rechecked every COMPRESS_MTIME_DELAY seconds)

  • full generated code ie.:

    <link rel="stylesheet" href="http://cdn.inprl.pl/CACHE/css/117f97d818b8.css" type="text/css">

Due to following cache usage django-compressor reduces number of reads to filesystem to 0. This is essential for page speed, because reading from memory is hundreds times faster than reading from filesystem. Also filesystem is very often bottleneck.

How it is stored in the cache?

django-compress is storing code in the cache using generated key. Key is generated from:

  • code in {% compress %}...{% endcompress %}
  • mtime of files mentioned in {% compress %}...{% endcompress %}

So those must be the same on all servers if you want to have consistent responses.

PS.

Please check constrains (like mtime) on your server and post here information if they match.

I will be fixing the same problem on my site probably next week, I will post additional details then.

Tomasz Wysocki
  • 11,170
  • 6
  • 47
  • 62
  • Thank you for your answer. Yes, reducing reads from file systems, that may be good but doesn't really speed things up that that much unless it's preventing some code from running and using resources. But that was not what I was asking for. I want to know what is stored in the cache and what it's used for. My guess is that it has something to do with the process of determining if new files should be generated. – demux Aug 31 '11 at 01:00
0

What you should do is to put all your compressed files on a Storage outside your computing instances that are behind the load balancer. For example, use Amazon S3 to store all your files on another subdomain than the rest of your application.

So http://myapp.com is pointing to your load balancer and http://s3.myapp.com is pointing to your storage, such as Amazon S3. You will not have to worry about the storing multiple different versions on different instances.

Here you can find a complete guide of how to setup Amazon S3, Gzip Compression and django-compressor with Django.

Marcus Lind
  • 10,374
  • 7
  • 58
  • 112
  • Lol, shameless selfpromotion. But yes, generally this is good advice, although it's not exactly an answer to the original question. In my specific case, back in 2011, the company I was working for was in business with a local hosting company that didn't provide this kind of content hosting. Here, have a cookie! +1 :D – demux May 06 '15 at 18:57