URL-based request rate limiting in Apache

Question

There are many rate limiting and QoS tools out there, but so far I have not been able to find one that meets my specific needs, as follows:

I have a web app running on Apache 2.2 and I want each of my customers to be limited to, say, 10000 requests per day. The limiting should not be based on the customer's IP address but on the customer's account, which can be extracted from the URL; e.g.: http://mywebsite.com/customer1/page1. Once they are over their limit, I would like requests from them to be "slowed down" to some predefined value, such as 15 requests per minute, perhaps by introducing a delay. If that's not possible, then returning an error like 503 Service Unavailable would be ok.

How can I do this with Apache? If there is no way to do it with Apache, then I'd be interested in suggestions for other tools, reverse proxies, etc.

UPDATE: The rate limiting needs to happen across a number of services (mod_dav_svn, mod_passenger, mod_wsgi) and must be high-performance.

Answer 1

I don't think you will be able to do this purely in Apache.

There are quite a few Apache modules that do bandwidth limiting but I don't think any of them will match what you have described.

mod_bw can limit bandwidth based on various parts of the request. It can limit based on IP address, file extension, mime type, file size and directory (based on my reading of the docs). You achieve the directory-based limiting by placing the module's directives inside a <Directory > block so I expect that they would work just as well inside a <Location > block. mod_bwshare seems to go inside <Directory > and <Location > blocks as well.

The configuration of these Apache modules go into your Apache configuration and they don't look up values from any external source, meaning you will have to restart Apache to change any of the values. If you want to only throttle the bandwidth once a customer has gone over their quota, you will need to edit the Apache config files and restart Apache every time any of them does so and do it again when their quota usage expires.

They way I would do this is to write a small application in whatever language you use that accepts a request for a file, checks in the database for whether the customer is over quota and then sends them the file at whatever rate is appropriate. The guy who wrote mod_bw explained how he achieved the throttling: he simply divides the file into small "chunks" (say 5KB per chunk) and then has a short sleep() in between echoing each chunk out to the client.

I would then use mod_rewrite to make translate the original file requests (that you said look like /customer1/page1) into something like /throttle.php?customer=customer1&file=page1.

The app can write to the database at the start of the download to indicate that it is currently processing a file and again at the end to indicate that it is now finished. This should enable you to stop one customer from hogging all your Apache children.

Answer 2

I just accepted @Ladadadada's answer of "It can't be done with Apache", but I thought I would list a couple of ways I am looking at working around this, for reference:

1. Write my own Apache module. It would use shared memory to keep track of all requests and which accounts they were to. These counts would be expired after 24 hours. Accounts that are over a lower limit would have each of their requests delayed for a certain length of time, and accounts that are over an upper limit would return a 503 status for all requests. The account name would be extracted from the URL using SetEnvIf.

2. Write a script (in Ruby) that would run as a daemon and tail the Apache access log. It would extract the account name from each logged request and track them similarly to #1, without the complexities of shared memory. It would also track all the IPs associated with a given account over the last 24 hours. When an account is over its limit, it would add NFQUEUE iptables rules for each IP associated with the account. The packets from these IPs would be captured by the Ruby script using Netfilter::Queue (in a second thread), where they would be delayed.