3

Our server got flooded with HEAD requests.

This resulted in a spike in tcp connections that made the server unable to connect to its mysql databases.

We make extensive use of the nginx ratelimiting, that works in combination with fail2ban flawlessly for any GET and POST requests. However, HEAD requests seem to not be picked up.

The fail2ban filter.d action we use is nginx-limit-req which seems to be a stock-config to detect when a nginx zone limit is reached.

As our webapp doesnt need HEAD requests, we have two options (aside from banning the individual IP addresses by hand)

  1. Is there a way to disable HEAD requests? It seems to be discouraged but is it possible nontheless?
  2. Would it be possible to make the nginx zone limiter recognize any GET/POST/HEAD request and limit it accordingly?

nginx.conf:

http {
  ...
  limit_req_zone "$http_x_forwarded_for" zone=web_zone:50m rate=2r/s;
  ...
}

server {
  ...
  fastcgi_buffers 16 16k;
  fastcgi_buffer_size 32k;

  location / {
    limit_req zone=web_zone burst=25;
    try_files $uri $uri/ @pretty-urls;
  }
  ...
}

jail.conf:

[nginx-limit-req]
enabled = true
filter = nginx-limit-req
action = custom-iptables-proxy
port = http,https
logpath = /var/log/nginx/*error*.log
findtime = 10
bantime = 3600
maxretry = 3

nginx-limit-req.conf:

[Definition]
ngx_limit_req_zones = [^"]+
failregex = ^\s*\[[a-z]+\] \d+#\d+: \*\d+ limiting requests, excess: [\d\.]+ by zone "(?:%(ngx_limit_req_zones)s)", client: <HOST>,
ignoreregex =
datepattern = {^LN-BEG}

example from the accesslog: enter image description here

netdata during the incident, normalized after the offenders ip was banned:

enter image description here

radonthetyrant
  • 183
  • 1
  • 6

1 Answers1

5

To disable HEAD requests in the Nginx configuration, just add the following into your server block:

if ($request_method = HEAD) {
    return 444;
}

The 444 HTTP status code is specific to Nginx and will drop all HEAD requests, so the connection fails for the client initiating the HEAD request. These attempted connections are still logged in Nginx logs, so you can ban the IP in question in your firewall configuration.