How to prevent Amazon ELB sits in front of RabbitMQ close connection with celery?

Question

1. ELB will auto close connection for 60 secs idling, with TCP connection switch to CLOSE_WAIT state
2. however, celery doesn't get noticed and keep publish task message
3. message will be kept in send buffer
4. when buffer is full, celery publishing call will be blocked.

Possible damages:

• Message in send buffer will be lost
• The blocking publishing call will be very harmful to single thread ioloop frameworks. e.g. Tornado

Answer 1

Solutions

• BROKER_TRANSPORT_OPTIONS = {'confirm_publish': True} to make celery wait for ack for each published message, if ack not receive, it will re-build connection and send again. Only apply to py-amqp (ref), performance downgrades.
• Celery-RabbitMQ Heartbeat to keep connection active and avoiding ELB's auto close connection. Add additional network overhead, heartbeat might not deliver to both end in bad network environment and cause this solution not working.