1

From time to time, I'm visited by "leechers" downloading an entire site (~2 GB) within a few hours, while the average visitor stays far below 50 MB. I'd like to setup a "byte-limit" per visitor (e.g. allowing max 100 MB per day and visitor).

I've tried mod_cband, which comes pretty close to my goal. Unfortunately I've managed only to establish a quote per VHost – i.e. if the quota was hit, the entire VHost becomes blocked. mod_cband also can manage quotas per remote IP – but for that I'd need to know those IPs in advance, which I do not.

I've also looked into mod_evasive, which I already use in slightly different context. But this only lets me limit the number of requests, and doesn't take the "volume" (bytes transferred) into account.

Are there any "ready-to-use" solutions available? If I've missed something with mod_cband, hints are welcome as well. If a solution cannot be bound to a VHost (but would apply server-wide), that would be acceptable as well (though per VHost is preferred).

Note I do not want to limit the bandwith (i.e. speed), neither limit simultaneously requests per IP; this is not about bandwidth but against "copy-cats".

Edit: I've just found Apache::Quota which seems to do pretty much what I want. But it a) requires to have mod_perl running (I'm not that familiar with Perl coding), and b) seems unmaintained (latest version is v0.04, dating back to 3/2007, and was intended for Apache 1.3 if I got that correctly).

Edit2: Solutions based on mod_security or iptables are welcome, too. So far, all I've found in this context was speed throtteling or limiting the amount of connections per remote IP, which is not what I'm after.

Edit3: Though I already found a solution for the underlying issue (see my answer below), I'm still interested in a solution to establish a "transfer quota per visitor and time", as described in my question – as my solution cannot be applied everywhere (see the "assumptions" described there).

Community
  • 1
Izzy
  • 349
  • 1
  • 5
  • 19
  • Possible duplicate of http://stackoverflow.com/questions/131681/how-can-i-implement-rate-limiting-with-apache-requests-per-second ? – Jakov Sosic Sep 21 '15 at 17:44
  • 1
    @JakovSosic nope I don't want to rate-limit "requests per second", but "data throughput per day". Actually I've checked with that question before (which is how I came to know about `mod_cband`). That question doesn't focus on the ***per visitor*** part, but is rather *per server/VHost*. (Apart from that, I'd VTC that for "too broad" ;) – Izzy Sep 21 '15 at 18:16

1 Answers1

0

Looks like I tapped into the XY trap – and completely missed a piece of software I'm already using for other things: Fail2ban. As the original intention was to stop leachers, that's easily done with a matching jail.

Assumptions

  • the site contains informational pages linking to local resources (e.g. ZIP or PDF files), which can easily be matched from the QUERY_STRING
  • "normal" visitors are browsing the pages, but only pick a few "resources"
  • a "leecher", processing all pages, would naturally hit more "resource files" in a short time frame than a "normal visitor" in all of the day. In my case, 50 hits/d would already be plenty for a "normal visitor".

Fail2Ban filter

The Fail2Ban filter makes use of those assumptions, matching only resources. This can be via file extensions (example above: .pdf or .zip), or by a path (e.g. with all resources being located in /downloads – but should make sure to not match the "normal pages". So here's an example filter (to be placed in /etc/fail2ban/filter.d/apache-leecher.conf):

[Definition]
# Match all our resources below /downloads:
failregex = ^ -.*"(GET|POST) /download.*"

# Alternatively, match all PDF/ZIP files
# failregex = ^ -.*"(GET|POST) .*\.(zip|pdf)"

(I've put the ZIP/PDF variant in as comment; you should only have one failregex in your filter file)

Fail2Ban Jail

Now for the corresponding jail, configured in /etc/fail2ban/jail.conf:

[apache-leecher]
# download no more than 100 files per hour, or get blocked for 6h (21600s)
enabled = true
port    = http,https
filter  = apache-leecher
logpath = /var/log/httpd/access_log
maxretry = 100
findtime = 3600
bantime  = 21600

Seems to work (already caught 2 candidates), but might need some fine-tuning. Anyone in a similar situation should be able to easily adapt the above to match the site.

Community
  • 1
Izzy
  • 349
  • 1
  • 5
  • 19
  • This is isn't working for me for some reason. I set it to maxretry = 2 to test it out, and it allows me to keep downloading more than 2. I changed the regex to the correct directory. – madprops Sep 14 '16 at 01:16
  • Make sure the path to your log file matches, and the `failregex` matches as well. Check the log for entries matching the `failregex` – and also check the status of `fail2ban`. Fail2ban sometimes reacts with a little delay, too. – Izzy Sep 14 '16 at 06:24