How to dynamically route message to queues in RabbitMQ

Question

I want to develop solutions that can dynamically route messages to different queues (more than 10000 queues). That's what I have so far:

• Exchange with type set to topic. So that I can route messages to different queues based on routing keys.
• 10000 queues that have routing key as #.%numberOfQueue.#. The %numberOfQueue% is simple numeric value for that queue (but it might be changed for more meaningfull ones).
• Producer producing message with routing key like that: 5.10.15.105.10000 which means that message should be routed to queue with keys 5, 10, 15, 105 and 10000 as they comform the patterns of that queues.

That's how it looks like from java client API:

String exchangeName = "rabbit.test.exchange";
String exchangeType = "topic";
boolean exchangeDurable = true;
boolean queueDurable = true;
boolean queueExclusive = false;
boolean queueAutoDelete = false;
Map<String, Object> queueArguments = null;

for (int i = 0; i < numberOfQueues; i++) {
    String queueNameIterated = "rabbit.test" + i + ".queue";
    channel.exchangeDeclare(exchangeName, exchangeType, exchangeDurable);
    channel.queueDeclare(queueNameIterated, queueDurable, queueExclusive, queueAutoDelete, queueArguments);

    String routingKey = "#." + i + ".#";
    channel.queueBind(queueNameIterated, exchangeName, routingKey);
}

That's how routingKey generated for all messages for queues from 0 to 9998:

private String generateRoutingKey() {
    StringBuilder keyBuilder = new StringBuilder();
    for (int i = 0; i < numberOfQueues - 2; i++) {
        keyBuilder.append(i);
        keyBuilder.append('.');
    }
    String result = keyBuilder.append(numberOfQueues - 2).toString();
    LOGGER.info("generated key: {}", result);
    return result;
}

Seems good. The problem is that I can't use such long routingKey with channel.basicPublish() method:

Exception in thread "main" java.lang.IllegalArgumentException: Short string too long; utf-8 encoded length = 48884, max = 255. at com.rabbitmq.client.impl.ValueWriter.writeShortstr(ValueWriter.java:50) at com.rabbitmq.client.impl.MethodArgumentWriter.writeShortstr(MethodArgumentWriter.java:74) at com.rabbitmq.client.impl.AMQImpl$Basic$Publish.writeArgumentsTo(AMQImpl.java:2319) at com.rabbitmq.client.impl.Method.toFrame(Method.java:85) at com.rabbitmq.client.impl.AMQCommand.transmit(AMQCommand.java:104) at com.rabbitmq.client.impl.AMQChannel.quiescingTransmit(AMQChannel.java:396) at com.rabbitmq.client.impl.AMQChannel.transmit(AMQChannel.java:372) at com.rabbitmq.client.impl.ChannelN.basicPublish(ChannelN.java:690) at com.rabbitmq.client.impl.ChannelN.basicPublish(ChannelN.java:672) at com.rabbitmq.client.impl.ChannelN.basicPublish(ChannelN.java:662) at com.rabbitmq.client.impl.recovery.AutorecoveringChannel.basicPublish(AutorecoveringChannel.java:192)

I have requirements:

• Dynamically choose from producer in which queues produce the messages. It might be just one queue, all queues or 1000 queues.
• I have more than 10000 different queues and it might be needed to produce same message to them.

So the questions are:

1. Can I use such long key? If can - how to do it?
2. Maybe I can achieve the same goal by different configuration of exchange or queues?
3. Maybe there are some hash function that can effectivily distinguesh destinations and collapse that in 255 symbols? If so, It should provide way to deal with different publishings (for example how to send to only queues numbered 555 and 8989?)?
4. Maybe there are some different key strategy that could be used in that way?
5. How else I can achieve my requirements?

Answer 1

I started using RabbitQM just a short time ago, hope I can help you nonetheless. There can be as many words in the routing key as you like, up to the limit of 255 bytes (as also described in e.g. RabbitMQ Tutorial 5 - Topics). Thus, the topics exchange does not seem to be appropriate for your use case.

Perhaps you can use a headers exchange in this case? According to the concept description:

A headers exchange is designed for routing on multiple attributes that are more easily expressed as message headers than a routing key. Headers exchanges ignore the routing key attribute. Instead, the attributes used for routing are taken from the headers attribute. A message is considered matching if the value of the header equals the value specified upon binding.

See here and here for an example. As I said, I just started with RabbitMQ, therefore, I don't know for sure whether this could be an option for you. If I have time later I try to construct a simple example for you.