3

I'm trying to setup HAProxy as a load balancer on a RabbitMQ cluster. I use a config like this one:

global
    log /dev/log    local0 debug
    log /dev/log    local1 debug
    chroot /var/lib/haproxy
    user haproxy
    group haproxy

defaults
    log     global
    retries 2
    timeout connect 5000
    timeout server 50000
    timeout client 50000

listen rabbitmq-cluster
    bind my.pu.blic.ip:5672
    mode tcp
    option tcpka
    option redispatch
    balance roundrobin
    server rabbit1 rabbit1:5672 check inter 5000 downinter 500
    server rabbit2 rabbit2:5672 check inter 5000 downinter 500
    server rabbit3 rabbit3:5672 check inter 5000 downinter 500

My issue is that clients that are supposed to keep connections to a rabbitmq server open have their connection closed every 50 sec. Is there an option I forgot, or should I just increase the timeout to some higher value to mitigate that issue?

Thanks!

axelcdv
  • 133
  • 1
  • 3
  • 1
    Could you do tcpdump of traffic relevant to this problem. Inspect if there is any activity on client side during 50 seconds. That should give you idea why connections are being dropped. Are you sure you need persistent connections there? Is re-initiating connection expensive in your opinion? Or are you trying to achieve persistant relations between those using RabbitMQ and specific backend server. – Hrvoje Špoljar Oct 30 '14 at 18:52
  • 1
    This is more detailed explanation and the solution: https://deviantony.wordpress.com/2014/10/30/rabbitmq-and-haproxy-a-timeout-issue/ – norcis Mar 12 '15 at 16:38

1 Answers1

4

It looks like your clients establish connection but after AMQP handshake there are no other activity (common case is when consumers are waiting for data from queue but queue empty).

From config you posted, timeout set to 50 sec, so closed connections after 50 seconds are expected when there are no network activity over that connection.

To deal with this you may increase timeout to reasonable for your use case value or enable heartbeat (see Hearbeat section), in addition look into Configuration manual (see hearbeat option description). In your case you have to change heartbeat value to something lower than 50 seconds, say timeout/3 rounded to ... say 15.

Normally when hearbeats used, hearbeat frames are sent after hearbeat interval after last connection activity. When three hearbeat frames failed (it takes 3 x hearbeat interval) connection should be closed by server or by client mandatory without any handshakes.

pinepain
  • 176
  • 7
  • That's more or less what happens, but it is a testing environment which doesn't see much traffic, so I guess I shouldn't really worry about that in production. But thanks for the suggestion about heartbeat! – axelcdv Oct 31 '14 at 11:10