Hitch rate limiting without Varnish

Question

Good morning,

Is there a way to implement per-site and per-IP rate limiting on Hitch before the request reach Varnish?

I tried to handle some DoS attack using Varnish rate limiting, but the DoS can overload the connection between Varnish and Hitch, before Varnish block the HTTP request.

Thank you.

Answer 1

Hitch has no rate limiting mechanism, but shouldn't be prone to high load either. Hitch can handle more than 100 Gbps on a single machine, so that shouldn't be the problem.

However, you mentioned a saturation on the connection between Hitch & Varnish.

Are these 2 components hosted on the same machine? If so, you can communicate over Unix Domains Sockets. This can reduce the stress on that connection, and would allow you to do rate limiting at Varnish's level.

Setting up a UDS backend connection in Hitch

In your hitch.conf file, you can set the backend as follows:

backend = "/var/run/varnish.sock"
write-proxy-v2 = on

This config will send all backend requests over the /var/run/varnish.sock Unix Domain Socket. It also makes sure the communication is done over the PROXY protocol, which makes life easier.

Setting up UDS connections in Varnish

In order for Hitch to send data to Varnish over UDS, Varnish needs to listen on that socket.

Here's an example where we add an additional listen port on /var/run/varnish.sock:

varnishd -a /var/run/varnish.sock,PROXY,user=varnish,group=varnish,mode=660 \
         -a :80 \ 
         -s malloc, 256m \
         -f /etc/varnish/default.vcl

The other parameters are just there to make the example seem more realistic. The main focus is -a.

Rate limiting

Varnish has a rate limiting module called vsthrottle. It is not installed by default, but available via https://github.com/varnish/varnish-modules.

Once you've compiled and installed the modules collection, you can use the following VCL code to rate limit incoming requests:

    vcl 4.0;
    import vsthrottle;

    backend default { .host = "192.0.2.11"; .port = "8080"; }

    sub vcl_recv {
        # Varnish will set client.identity for you based on client IP.

        if (vsthrottle.is_denied(client.identity, 15, 10s, 30s)) {
            # Client has exceeded 15 reqs per 10s.
            # When this happens, block altogether for the next 30s.
            return (synth(429, "Too Many Requests"));
        }

        # There is a quota per API key that must be fulfilled.
        if (vsthrottle.is_denied("apikey:" + req.http.Key, 30, 60s)) {
                return (synth(429, "Too Many Requests"));
        }

        # Only allow a few POST/PUTs per client.
        if (req.method == "POST" || req.method == "PUT") {
            if (vsthrottle.is_denied("rw" + client.identity, 2, 10s)) {
                return (synth(429, "Too Many Requests"));
            }
        }
    }