1

I am currently adding rate limiting to http and mqtt services in haproxy. These are working, but I also want to show the blocked requests in the logs. On the http frontend I return a 429 and on the mqtt I close the connection on blocked requests.

But when I enable logs, ALL logs are shown.

global
    log 127.0.0. local notice

defaults
    log global
    option httplog



#sticky tables
backend st_http
    stick-table type ipv6 size 100k expire 10s store http_req_rate(10s)

backend st_mqtt
    stick-table type ipv6 size 100k expire 10s store conn_rate(10s)



frontend public-https
    log stdout format short daemon warning

    #rate limiting
    http-request track-sc0 src table st_api_requests
    http-request deny deny_status 429 if { sc_http_req_rate(0) gt 100 }

    # ...

listen mqtt:
    mode tcp
    option tcplog
    log stdout format short daemon warning

    # rate limiting
    tcp-request connection track-sc0 src table st_mqtt_connections
    tcp-request connection reject if { sc_conn_rate(0) gt 2 }

    # ...

How can I only show faulty requests, i.e. all 4xx and 5xx requests on http and all blocked connections on mqtt?

// My current tests only show all log output of a service or none depending on the loglevel. How can I make this more selective?

//edit2 added acls for rate limiting

Frankie777
  • 11
  • 3
  • can you post the acl stuff for the rate limiting? – Pedro Rodrigues Nov 04 '20 at 14:30
  • 1
    sure. I've updated the rate limiting. But I do not understand how this will help with the logging. – Frankie777 Nov 05 '20 at 09:51
  • As long as you reject the tcp connections I do not think you can log them. Instead of rejecting the TCP connection, deny it with a 429 as you did in the public-https section. That way 429s from mqtt are logged and returned to the client. You may even add special purpose headers to indicate the reason for the 429. – Pedro Rodrigues Nov 05 '20 at 10:58
  • Because you can't log rejected TCP connections. https://www.mail-archive.com/haproxy@formilux.org/msg10795.html – Pedro Rodrigues Nov 05 '20 at 10:59
  • I can accept, that I cannot log rejected TCP connections. But returning an HTTP 429 response over an MQTT connection is not valid. Regarding the logging of 4xx/5xx HTTP requests: How can I proceed there? If I log all requests, the log will fill up quite fast. – Frankie777 Nov 05 '20 at 14:11
  • ok. I see your point there, I wrongly assumed the HTTP protocol for some reason. Check [dontlog-normal](https://cbonte.github.io/haproxy-dconv/2.1/configuration.html#4-option%20dontlog-normal) and [log-separate-errors](https://cbonte.github.io/haproxy-dconv/2.1/configuration.html#option%20log-separate-errors) – Pedro Rodrigues Nov 05 '20 at 14:24

1 Answers1

0

Check dontlog-normal and log-separate-errors. If you need more than those can offer I suggest looking at some logging infrastructure.

HAProxy won't log rejected TCP connections. You're stuck with that, rate limit at the mqtt level and perhaps use HAProxy to load balance a cluster.

Pedro Rodrigues
  • 111
  • 3