0

I have a web service endpoint that I am building where people will post an xml file to, and it will really get pounded with over 1K requests per second.

Now they are sending in these xml files via http post, but a good majority of them will be rate limited.

The problem is, the rate limiting will be done by the web application by looking up the source_id in the xml, and if it is over x requests per minute, it will not be processed further.

I was wondering if I could do rate limit checking earlier in the processing somehow and thus save the 50K file going threw the pipeline to my web servers and eating up resources.

Could a load balancer make a call out to verify rate usage somehow?

If this is possible, I could maybe put the source_id in a host header so even the XML file doesn't have to be parsed and loaded into memory.

Is it possible to just look at host headers and not load up the entire 50K xml file into memory?

I really appreciate your insights as this takes more knowledge of the entire tcp/ip stack etc.

Blankman
  • 2,891
  • 10
  • 39
  • 68
  • You can rate limit using the Host header (i.e. virtual hosts) on Nginx with the [HttpLimitReqModule](http://wiki.nginx.org/NginxHttpLimitReqModule). Something similar may be possible with HAproxy, as described in the [Server Fault blog](http://blog.serverfault.com/2010/08/26/1016491873/). – cyberx86 Nov 26 '11 at 20:08

3 Answers3

1

There are two basic types of possibilities:

1) You are dealing with hostile attackers who will not cooperate with your design. In this case, why do you care what Host header they're sending? Rate-limit them by IP.

2) You are dealing with cooperating clients who are imposing more load than you want. In this case, just accept the request and send them a reply that tells them to back off.

If you have a more complicated situation that doesn't fall into one of these two simple categories, the best solution will likely depend on what that situation is.

David Schwartz
  • 31,449
  • 2
  • 55
  • 84
  • My situation would be #2. I was hoping there was a way to prevent that request from using up resources on my web servers, there will be a really high rate of duplicates that the client cannot control. – Blankman Nov 27 '11 at 17:11
  • Why can't the client control the load it's placing on the server? – David Schwartz Nov 27 '11 at 22:48
  • Things are being sent based on certain events, and the logic of controlling load cannot be on their end as it may change etc. and the client library cannot be updated that often which is out of my control. – codecompleting Nov 30 '11 at 16:51
  • @codecompleting The idea is that the server would tell the client if it's placing too much load on the server. If it changes, you tell the client that it has changed. You won't have to update the client library because it will contain fixed logic to respect the rate limits that the server commands. – David Schwartz Nov 30 '11 at 17:56
0

You can block it via apache with similar modules. Or if you want to cut it out earlier you can use iptables string matching to block things via a regex.

n8whnp
  • 1,326
  • 7
  • 9
0

This is what the tc command/utility is for - it takes a bit of working out, but it's worth the effort. I use it to rate limit transfers to a server in a DMZ off our corporate LAN.

Here's a few links that got me on track...

Tc Filter - Port Ranges Calculate Mask Value: http://mailman.ds9a.nl/pipermail/lartc/2007q4/021739.html

Rate limiting a single host or netmask: http://lartc.org/howto/lartc.ratelimit.single.html

http://www.linuxquestions.org/questions/linux-networking-3/limit-bandwidth-rate-for-scp-using-tc-htb-linux-825684/

Linker3000
  • 668
  • 1
  • 5
  • 14