haproxy BADREQ errors

Question

I am seeing errors similar to the following in my haproxy logs:

Jul 18 17:05:30 localhost haproxy[8247]: 188.223.50.7:51940 [18/Jul/2011:17:05:24.339] http_proxy_ads http_proxy_ads/<NOSRV> -1/-1/-1/-1/6001 408 212 - - cR-- 100/89/0/0/0 0/0 "<BADREQ>" 
Jul 18 17:05:30 localhost haproxy[8247]: 188.223.50.7:51943 [18/Jul/2011:17:05:24.341] http_proxy_ads http_proxy_ads/<NOSRV> -1/-1/-1/-1/6000 408 212 - - cR-- 99/88/0/0/0 0/0 "<BADREQ>" 

etc...

So far I have tried to increase the client timeout (to 6 seconds from 3), and increase the http request buffer from 16k to 32k. The errors still appear.

Can anyone give me guidance on what to look for here?

Answer 1

A Preconnect from a browser could lead to BADREQ too if the browser is not using all connections. For example when a user is downloading only one file per browser.

That means there are two possible causes for BADREQ with cR-- or CR-- (verified with HAProxy v1.5-dev24):

1. Unused connection: That means for HTTP(S) a client connected per TCP but no HTTP request header was sent until from timeout http-request (CR--) or the client was closing the connection again (cR--). Cause: Unused connection from a preconnect of a normal client or loadbalancer or from a scan.
2. Bad Request. A client was sending a bad request. These errors should be visible per stats socket (see previous answer from womble).

Most modern browsers like Firefox or Chrome are doing a preconnect. I was seeing that Firefox or Chrome were opening always at least 2 connections even if the browser is doing only one request like downloading a file (for example only downloading http://cdn.sstatic.net/serverfault/img/favicon.ico)

Increasing the value of timeout http-request in your HAProxy configuration can help to reduce such log entries for unused connections just because a higher value means a higher chance that the connection will be used from a client, but you are also risking that your server cannot handle all open (idle) connections anymore. If you are using another loadbalancer like Amazon ELB in front of HAProxy, check that this timeout in HAProxy is matching with the loadbalancer, because they could use preconnect too.

For unused connections you can use option dontlognull in HAProxy to disable this log entries. Quote from HAProxy Docu for this option:

It is generally recommended not to use this option in uncontrolled environments (eg: internet), otherwise scans and other malicious activities would not be logged.

Answer 2

=> The client never completed its request, which was aborted by the
   time-out ("c---") after 5s, while the proxy was waiting for the request
   headers ("-R--").  Nothing was sent to any server, but the proxy could
   send a 408 return code to the client.

Solution: change "timeout http-request" to 20s ore more instead your 5s.

Answer 3

Just wasted a day on this issue. We found out that some request headers were too large. 8k is the default max size in HAProxy (all headers combined) and our company loves cookies. In our case, about 8% of the request headers were too large and therefore truncated after the 8092th byte.

From the docs:

If HTTP request is larger than (tune.bufsize - tune.maxrewrite), haproxy will return HTTP 400 (Bad Request) error. Similarly if an HTTP response is larger than this size, haproxy will return HTTP 502 (Bad Gateway).

We updated the haproxy.cfg file with the following values:

global
  # request limit is (bufsize - maxrewrite), our desired limit is 16k (8k is default)
  tune.maxrewrite  16384
  tune.bufsize     32768

Hope it helps!

Answer 4

BADREQ just means that the client sent a bad request; in HTTP mode, it could mean that the client stuffed up, and there's nothing you can do about it. To see what the exact error was, connect to the stats socket (using socat) and run show errors. Chances are it's someone trying to run an exploit on some other webserver, so you can ignore it.