RabbitMQ let consumer receive more tasks before completing earlier task

Question

Based on my understanding of RabbitMQ/AMQP I see an unfortunate tradeoff.

The usual workflow is:

1. Producer produces
2. Queue delivers to consumer
3. Consumer acks and does not receive more tasks until the ack
4. Queue considers message processing complete

So the problem is the ack couples two things: saying "I can accept more work," and saying "I'm done with this task." Problem is! My consumer machines are profiled to process 50 tasks in parallel and each task takes up to about 30 seconds. But if one task dangles or fails - it needs to be redelivered. How to accomplish this?

Answer 1

Use

channel.basicQos(prefetchCount);

Where prefetch count = number of tasks I can handle without acknowledging.

Documented e.g. here.