5

For a few months now one of our shared hosting servers has been persistently and constantly hammered by "POST /" requests from what must be hundreds of thousands of individual IPs. On a number of occasions this has overwhelmed the server and led to a denial of service-type outage. The target domain is pretty boring (a small Estate Agent) so whilst this appears to be malicious I can't understand motive of this long-running and 99% unsuccessful attack.

A typical request (taken from TCPDUMP) looks something like this:

POST / HTTP/1.1
Accept: */*
Accept-Language: en-us
Content-Type: application/octet-stream
Content-Length: 570
User-Agent: Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1)
Host: xxx.co.uk
Connection: Keep-Alive
Cache-Control: no-cache

2+cIPSyYVJFqB9xPFzWoLj9seNyEKIcuIJz/yfkc9tVP+orXgjDk8ywglufcXsMz
bVP4XLcowz/fQtsn2kceQEj/EaEWx/GEbcC3eTbCbTube0suAfEGje3qISKQJ+ka
HaChqun3whii3OTh7vCayGV72lh4raLRandKC5g/73wgQ9Jzh2OLIzNvsiEMSJco
yG+4i35XJMvX7ovx8qJkyByHUIeE5G5M2Kp97O4sOT4jTAK2y/KAMjf6oFgtAJhI
K4/HdcnyfNdI3/4RJXlrSfhUQAc+qhGMEL7AZdtzgRub7lnu+hbuPGZvS3rF1MvL
WK1q4mrnZr0Q3m0bWkzsMZCndQ7fqOBafchjprhn4JKPsjO+upRm2m+irvmJjqnl
sDiR3fnD6pzbWyLTm2qonMJPCll3p6zg06gEfIaW04t9r89/PdHgz8AU8nzO4BX8
qwTG6dSjgbowHyJQmud8Ro+ZT+gHfw/YQUrBqKm7RoFmfJzUoOCKaP1LTwHfI1Gc
E+L8bwQV6ztKBwVn2NqbE83SAXYr9E0QkpaxGg==

We haven't been able to determine what's in the POST request as it looks like garbage, but I'm not sure its relevant. It's not base64 encoded.

To reduce the amount of bandwidth being used up by responses to this request we have banned the use of POST requests in the Apache2 configuration:

<Location />
    <Limit POST>
        Order deny,allow
        Deny from all
    </Limit>
</Location>

This restricts the response size to just a simple 403 Forbidden message, rather than the client's usual homepage.

To try and block the IPs doing this we've tried piping the access log, filtering for the POST request, and feeding this directly into iptables:

tail -f /var/www/vhosts/xxx.co.uk/statistics/logs/access_log | grep "POST / " | awk '{print $1}' | xargs -I{} iptables -A INPUT -s {} -j DROP

This works well and reduces the effect of the problem, but it is relentless and we usually have to clear the iptables rule set when it reaches 50-60k due to iptables/kernel problems. It's not a solution as I can't just leave this running for a few weeks until whoever is responsible gets the message and gives up.

We've turned off KeepAlive for this particular VirtualHost too to keep the number of occupied Apache workers to a minimum which has helped, but it's not a solution.

Does anybody have any better ideas on how to blackhole these requests, on a scale of hundreds of thousands of remote IPs, or to reduce the impact on Apache to the absolute minimum? The best I can do at the moment is configuring it to send a 403 Forbidden, combined with IP-blocking for a few hours...

Thanks!

Richard Maynard
  • 51
  • 4
  • do you have wordpress installed? – that guy from over there Oct 30 '13 at 13:00
  • Sure the real estate agency hasn't ticked off the competition who hired a botnet? – Engineer2021 Oct 30 '13 at 13:04
  • The site is just a bunch of html files, there's no server-side scripts or anything exploitable at all! I wondered about botnet employed by their competition, but it's been going on for months and almost completely unsuccessful. If they wanted to take the site/server offline it wouldn't be hard... – Richard Maynard Oct 30 '13 at 13:52

1 Answers1

2

if you have root-access to that machine you could deploy snort/suricata with a limited ruleset, e.g. detect and block POST - requests.

pro:

  • works on network-level
  • ips can be blocked for a certain ammount of time

con:

  • not-so-easy to setup and maintain, should be done by someone who knows how/what to do

easier: setup nginx infront of your apache and process your 403 from there, while passing valid requests to apache

pro:

  • easy setup
  • can handle more requests than apache
  • limit-rate might be used on a ip-level

con:

  • must be tested
that guy from over there
  • 1,355
  • 10
  • 11