Kombu, RabbitMQ: Ack message more than once in a consumer mixin

Question

I have stumbled upon this problem while I was documenting Kombu for the new SO documentation project.

Consider the following Kombu code of a Consumer Mixin:

from kombu import Connection, Queue
from kombu.mixins import ConsumerMixin
from kombu.exceptions import MessageStateError
import datetime

# Send a message to the 'test_queue' queue
with Connection('amqp://guest:guest@localhost:5672//') as conn:
    with conn.SimpleQueue(name='test_queue') as queue:
        queue.put('String message sent to the queue')


# Callback functions
def print_upper(body, message):
    print body.upper()
    message.ack()    

def print_lower(body, message):
    print body.lower()
    message.ack()


# Attach the callback function to a queue consumer 
class Worker(ConsumerMixin):
    def __init__(self, connection):
        self.connection = connection

    def get_consumers(self, Consumer, channel):
        return [
            Consumer(queues=Queue('test_queue'), callbacks=[print_even_characters, print_odd_characters]),
        ]

# Start the worker
with Connection('amqp://guest:guest@localhost:5672//') as conn:
    worker = Worker(conn)
    worker.run()

The code fails with:

kombu.exceptions.MessageStateError: Message already acknowledged with state: ACK

Because the message was ACK-ed twice, on print_even_characters() and print_odd_characters().

A simple solution that works would be ACK-ing only the last callback function, but it breaks modularity if I want to use the same functions on other queues or connections.

How to ACK a queued Kombu message that is sent to more than one callback function?

Answer 1

Solutions

1 - Checking message.acknowledged

The message.acknowledged flag checks whether the message is already ACK-ed:

def print_upper(body, message):
    print body.upper()
    if not message.acknowledged: 
        message.ack()


def print_lower(body, message):
    print body.lower()
    if not message.acknowledged: 
        message.ack()

Pros: Readable, short.

Cons: Breaks Python EAFP idiom.

2 - Catching the exception

def print_upper(body, message):
    print body.upper()
    try:
        message.ack()
    except MessageStateError:
        pass


def print_lower(body, message):
    print body.lower()
    try:
        message.ack()
    except MessageStateError:
        pass

Pros: Readable, Pythonic.

Cons: A little long - 4 lines of boilerplate code per callback.

3 - ACKing the last callback

The documentation guarantees that the callbacks are called in order. Therefore, we can simply .ack() only the last callback:

def print_upper(body, message):
    print body.upper()


def print_lower(body, message):
    print body.lower()
    message.ack()

Pros: Short, readable, no boilerplate code.

Cons: Not modular: the callbacks can not be used by another queue, unless the last callback is always last. This implicit assumption can break the caller code.

This can be solved by moving the callback functions into the Worker class. We give up some modularity - these functions will not be called from outside - but gain safety and readability.

Summary

The difference between 1 and 2 is merely a matter of style.

Solution 3 should be picked if the order of execution matters, and whether a message should not be ACK-ed before it went through all the callbacks successfully.

1 or 2 should be picked if the message should always be ACK-ed, even if one or more callbacks failed.

Note that there are other possible designs; this answer refers to callback functions that reside outside the worker.