Good way to choose between backends by Varnish depending on requested file existence

Question

I have the installation with Apache2, nginx and Varnish above them. Varnish receives the request, determines what backend is it for (static goes to nginx, dynamic goes to Apache), fetches the response from backend and caches it if needed.

Amongst other things Apache generates thumbnails. It creates thumbnails using some specific logic and it is quite expensive for processor. Thus Apache saves thumb to disk in order next time the request for this thumbnail will be received Varnish could redirect it directly to nginx.

And here is a problem: Varnish has no way to check if file exists somewhere in file system and that's why it don't know if nginx backend can be used or Apache should generate thumbnail first.

The workaround which I've currently adopted is quite simple but ugly:

Varnish receives the request for thumbnail;
Varnish sends request to nginx;
If nginx can't find thumbnail file it replies with 404 error and Varnish redirects request to Apache.

While this algorithm seems to be simple in reality it is not. Following implementation is needed in VCL configuration file:

In vcl_recv always assume that thumbnail exists and all request must be redirectedto nginx if it is not requested from some special host;
In vcl_fetch catch object HTTP status and if it is 404 and resource is thumbnail than rewrite host to special one and restart the process:

VCL:

if( obj.status == 404 ) {
    if(req.url ~ "^thumb/") {
        set req.http.host = "thumb_generator.site.com";
        set req.url = regsub(req.url, "/thumb/(filename)", "thumb_gen.php?filename=\1");

        restart;
    }
}

Maybe there are some better ways to solve this problem? I know that Varnish supports C in VCL, maybe it is better to check file existence using C-code?

Please use the tag varnish-vcl instead of vcl, since vcl is a popular tag already in the Delphi community. A mod has retagged all varnish vcl tags today. — Warren P, Nov 02 '11 at 23:35

WASD42 · Accepted Answer · 2011-11-02T15:24:45.900

OK, if anyone is interested in solution I found a new one with inline C in VCL config. First of all we must add a function to check if file exists (add somewhere in the top of your file outside of any function):

C{
#include  <stdio.h>
#include  <stdlib.h>

int exists (char *fname)
{
    FILE *file;
    if (file = fopen(fname, "r"))
    {
        fclose(file);
        return 1;
    }

    return 0;
}
}C

I know that there are better ways to check if file exists, but major headers are not available inside VCL :/

Then in vcl_recv subroutine add following code:

C{
    if( exists("/local/file/path") == 1 ) {
        VRT_l_req_backend(sp, VCL_conf.director[1]);
    } else {
        VRT_l_req_backend(sp, VCL_conf.director[2]);
    }
}C

Works like a charm.

Charms don't work. But good solution! If you are trying to reduce the latency, you might consider using fstat() on the file instead, which I believe performs less i/o on the disk. It does not actually open the file, it just confirms that the file is there from the directory entry: http://linux.die.net/man/2/fstat — JasonSmith, Jun 26 '15 at 14:36

score 2 · Answer 2 · answered Nov 02 '11 at 19:28

You're using Varnish for something it wasn't made for. It indeed has no possibility to check if a file exists, because it was not made to serve files in the first place. Varnish just temporarily caches back-end responses in file or memory storage.

Do you really need varnish in this setup? Wouldn't it make a lot more sense to have nginx check if the file exists or else forward it to your processor?

score 2 · Answer 3 · answered Feb 04 '12 at 11:39

2

Why not use Nginx's try_files directive to silently proxy the request to Apache on a 404? It would seem more logical, to me at least.

answered Feb 04 '12 at 11:39

uknzguy

409
3
4

score 1 · Answer 4 · answered Feb 12 '14 at 20:51

You probably will be interested in libvmod-utils. You will have access to what WASD42 suggested but directly with a VMOD (cleaner, and should have latest improvements & advices applied) instead of inline-C.

Inline-C is good because you can do whatever you need, but sometime having a VMOD is cleaner and will allow to not disrupt your VCL language.

Below an example of VCL implementing your needs :

import utils;

sub vcl_recv {
    if(req.url ~ "^/thumb/") {
        if (utils.exists("/srv/www/static/" + req.url)) {
            set req.backend = nginx;
        else {
            set bereq.url = regsub(req.url, "/thumb/(filename)", "thumb_gen.php?filename=\1");
            set req.backend = apache;
        }
    }
}

Good way to choose between backends by Varnish depending on requested file existence

4 Answers4