1

I've recently found out RabbitMQ feature that allows you to delay messages and it works great although I couldn't find any examples similar to what I need:

Let's say there are 3 types of messages: A, B and C. We've got 2 delay_queues with 1 hour and 2 hours 'x-message-ttl values. There are also 3 types of destination_queues - each for specific message type.

What I would like to achieve is after the message in one of the delay_queues reaches its TTL it's going to be routed to one of the destination_queues depending on its type. Something like this:

enter image description here

Is this even possible using RabbitMQ message properties? Any ideas? My code sending messages to the delay queue (after expiration they're sent to hello queue):

#!/usr/bin/env python
import pika

connection = pika.BlockingConnection(pika.ConnectionParameters(
               'localhost'))

channel = connection.channel()
channel.confirm_delivery()
channel.queue_declare(queue='hello', durable=True)

channel.queue_bind(exchange='amq.direct',
                   queue='hello')

delay_channel = connection.channel()
delay_channel.confirm_delivery()

delay_channel.queue_declare(queue='hello_delay', durable=True,  arguments={
  'x-message-ttl' : 3600000,
  'x-dead-letter-exchange' : 'amq.direct',
  'x-dead-letter-routing-key' : 'hello'
})

while 1 :
        delay_channel.basic_publish(exchange='',
                      routing_key='hello_delay',
                      body="test",
                      properties=pika.BasicProperties(delivery_mode=2))
        print "Sent to delay queue"
Herwish
  • 93
  • 1
  • 9

3 Answers3

3

When messages expire because they reach the imposed TTL, they can be redirected to a dead-letter exchange, that's what I think you are using implicitly to move messages to a further queue after they are expired.

You can select different destination queues by using the original routing-keys of the messages, or eventually "CC" and "BCC" message headers.

Sigi
  • 4,826
  • 1
  • 19
  • 23
  • can you provide an example of code with dead-letter-routing-key that belongs to the message not to the queue declaration? – Herwish May 28 '15 at 08:33
  • I think you are right (answer corrected): x-dead-letter-routing-key applies to queues not messages. I will extend the answer and check it deeper later. However you can use the original message routing-key for sure and "CC" and "BCC" headers (you can let these destinations be ignored in the original exchange, by not binding any queue matching these routing-keys, and instead be properly used in the dead-letter exchange only) – Sigi May 28 '15 at 09:45
  • 1
    However in this case you must not use any 'x-dead-letter-routing-key', or it will override the original message destinations. – Sigi May 28 '15 at 09:55
2

Okay so I've managed to find a solution. Not sure if it's the best one but it works.

  1. I created two exchanges: DELAY_EXCHANGE and ROUTER_EXCHANGE
  2. I binded DELAY_EXCHANGE to the delay_queue (with all of the routing_keys I use)
  3. Delay queue is set with x-dead-letter-exchange: ROUTER_EXCHANGE and x-message-ttl: 14000
  4. I binded ROUTER_EXCHANGE to all of the queues (A,B,C) with the corresponding routing_keys.

This way while sending (pushing) the message I don't specify the queue, just exchange and th routing_key:

    delay_channel.basic_publish(exchange='delay_exchange',
                  routing_key='helloC',
                  body="test",
                  properties=pika.BasicProperties(delivery_mode=2))

Message is pushed to the DELAY_EXCHANGE which directs it to the delay_queue, where it waits for its TTL. When the message expires it's redirected to the ROUTER_EXCHANGE which analyzes its routing_key and redirects it to one of the destination queues. Awesome.

Herwish
  • 93
  • 1
  • 9
0

You could use the new RabbitMQ Delayed Message Exchange https://github.com/rabbitmq/rabbitmq-delayed-message-exchange/

old_sound
  • 2,243
  • 1
  • 13
  • 16