1

We have an Nginx / PHP-FPM behind a load balancer which periodically gets scraped. Many of the scrapers hit our application's 404 page quiet hard.

Is there a way in Nginx, possibly with the rate-limiting module or a different module, to block all traffic based on x-forward-for after hitting 404 errors at over 100 requests per minute?

The documentation I found for limit-req-module looks like it is based on resource rather than on page return status.

There is enough traffic, that each node is seeing enough 404 traffic that they shouldn't need to communicate about who to collectively block.

rrehbein
  • 488
  • 4
  • 9
  • Perhaps fail2ban – Hyppy May 15 '15 at 19:29
  • As the IP is in the x-forward-for, would fail2ban be able to update nginx's configuration? – rrehbein May 15 '15 at 19:41
  • I'd say the normal method of creating a temporary firewall rule would be sufficient. – Hyppy May 15 '15 at 19:42
  • Normal firewall rules would block the traffic from the load balancer, not just the bot. It doesn't look like there is a way to setup ip based filtering at the load balancer. – rrehbein May 18 '15 at 13:12
  • It looks like I found an answer http://www.michelem.org/2014/08/01/nginx-fail2ban-blocking-ip-behind-aws-load-balancer/ – rrehbein May 18 '15 at 13:30

2 Answers2

3

You can certainly do this in nginx with the limit_req_zone module.

In nginx.conf setup a zone:

limit_req_zone $binary_remote_addr zone=one:1000m   rate=100r/m;

You can also use X_Forwarded_For instead of binary_remote_addr too.

In your site config location block just reference the zone:

limit_req zone=one burst=10 nodelay;

doc: http://nginx.org/en/docs/http/ngx_http_limit_req_module.html

masegaloeh
  • 18,236
  • 10
  • 57
  • 106
d_turner
  • 56
  • 5
0

You can use fail2ban and use following jail with nginx to rate limit misconfigured bots

cp /etc/fail2ban/jail.conf /etc/fail2ban/jail.local and add this at bottom

[nginx-req-limit]

enabled = true
filter = nginx-req-limit
action = iptables-multiport[name=ReqLimit, port="http,https", protocol=tcp]
logpath = /var/log/nginx/*error.log
findtime = 600
bantime = 7200
maxretry = 10

references: https://rtcamp.com/tutorials/nginx/fail2ban

Harshad
  • 121
  • 2
  • I am not sure about nginx but on apache you must make sure that you actually log the ip of the source, not the load balancer, you would not want the load balancer ip to get blocked. – Dennis Nolte May 19 '15 at 13:03
  • iptables does not see the source IP, as it is in the x-forward-for, not in the tcp header. – rrehbein May 19 '15 at 13:23