3

I have a number of EC2 servers on AWS running apache behind a load balancer (ELB). Every now and then some IP address abuses the API hosted on the EC2 servers and causes a denial of service. I have no access to the load balancer so I need to block access at the server's level.

I changed the apache access log to display IP's based on the X-Forwarded-For header provided by the load balancer (otherwise it just displays the load balancer's IP), so I can identify these IP's and block them (again by specifying the X-Forwarded-For) with something like:

<Directory api_dir>
                SetEnvIF X-FORWARDED-FOR "1.1.1.1" DenyIP
                Order allow,deny
                allow from all
                deny  from env=DenyIP
</Directory>

However, this still means that I need to manually handle every attack, and my server suffers some downtime as a result.

What is the recommended way to automatically block attacks of repeated HTTP calls, based not on IP but on the Forwarded-For header coming from the load balancer.

hillel
  • 131
  • 1
  • 4

2 Answers2

2

You can do this with mod_evasive combined with mod_rpaf.

The former lets you limit by IP address (to stop DOS attacks, for example), the latter allows you to make the X-FORWARDED-FOR address appear as the IP, as discussed in the comments of the answer to this question.

(If mod_rpaf doesn't work for you, you'd have to hack the source of mod_evasive.)

Community
  • 1
Keith
  • 4,637
  • 15
  • 25
  • Are you familiar with this thread http://serverfault.com/questions/233092/mod-evasive-behind-haproxy ? It sounds like there's a serious problem with this method, but I'll look into it. – hillel Aug 21 '13 at 17:24
  • Mmmmm, looks like someone needs to fix mod_evasive. Some of the alternatives discussed in this thread deserve attention, too: http://www.dslreports.com/forum/r20353461-apache-modevasive-sucks-as-antiddos-protection – Keith Aug 21 '13 at 21:23
2

I wrote up an extended blog post about how to do this with Apache 2.4 using OSSEC (a FOSS host-based intrusion detection system). Here's the TLDR:

  1. Use mod_remoteip to log client IPs in place of your load-balancer IPs.
  2. In your vhost config, use an IncludeOptional directive inside a RequireAll directive to source in a directory that will contain other mod_authz_host directives, written to by OSSEC.
  3. Create a custom active-response script for OSSEC that will add a file with a mod_authz_host directive like Require not ip 1.2.3.4 to the above directory for each IP address from which OSSEC detects malicious activity.
  4. Configure OSSEC to start using the above active-response script -- I'd recommend copying the settings from the command and active-response config blocks of OSSEC's default host-deny active-response, which blocks an IP address for 10 minutes if it triggers an alert at or above level 6.
  5. Add some custom rules to OSSEC's local_rules.xml definition file for HTTP requests logged to your Apache access log that will trigger the above active response when matched with X frequency in Y timeframe (like say more than 50 times in 10 seconds, or whatever you consider abusive).

If you aren't already using OSSEC (or some other HIDS), I'd recommend it heartily -- it's very easy to set up, and provides a lot of functionality right out of the box. DigitalOcean has a good OSSEC install tutorial for Ubuntu 14.04 (which you can follow along pretty well even for other linuxes). You can just install it in "local" mode on each of your servers to get started, and later on rearrange it to use its fancy client-server architecture.

Also note this technique requires automatically restarting Apache every time an IP is added to the block list, which may be a non-starter for some.

Justin Ludwig
  • 1,276
  • 9
  • 9
  • Thanks Justin, I can't accept the answer since I no longer have that configuration to verify, but looks comprehensive. – hillel May 12 '15 at 16:52