2

Let's consider a system where thousands of clients data is published to a RabbitMQ exchange (client_id is known at this stage). Exchange routes them to a single queue. Finally, messages are consumed by a single application. Works great.

However, over time, the consuming application becomes a bottleneck and needs to be scaled horizontally. The problem is the system requires that messages considering particular client are consumed by the same instance of the application.

I can create lots of queues: either one per client or use a topic exchange and route it based on some client_id prefix. Still, I don't see an elegant way how to design the consumer application so that it can be scaled horizontally (as it requires stating queues that it consumes explicitly).

I'm looking for RabbitMQ way for solving this problem.

michal.dul
  • 135
  • 1
  • 2
  • 10

1 Answers1

4

RabbitMQ has x-consistent-hash and x-modulus-hash exchanges that can be used to solve the problem. When these exchanges are used, messages get partitioned to different queues according to hash values of routing keys. Of course, there are differences between x-consistent-hash and x-modulus-hash in the way how partitioning is implemented, but main idea stays the same - messages with the same routing key (client_id) will be distributed to the same queue and eventually should be consumed by the same application.

For example, the system can have the following topology: every application can define an exclusive queue (used by only one connection and the queue will be deleted when that connection closes) that is binded to the exchange (x-consistent-hash or x-modulus-hash).

In my opinion, it is a good idea to have a distributed cache layer in this particular scenario, but RabbitMQ provides the plugins to tackle this kind of problems.

Sergii Zhevzhyk
  • 4,074
  • 22
  • 28
  • Many thanks for your reply. Your answer solves my problem partially. The part I still don't know how to solve in the stateful approach is how to create a consumer application so that a new instance can "take over" some of the queues/clients from existing instance. Example: using `x-cosistent-hash` with n queues and one instance of the consumer I want to add a second consumer. Here's what I'm looking for: how can the second container take over n/2 queues? – michal.dul Jul 17 '18 at 14:28
  • 1
    @michal.dul With consistent hashing, you do not deal with traffic redistribute this way. You can just redistribute traffic among all queues with weights. From the doc: "So, if you wish for queue A to receive twice as many routing keys routed to it than are routed to queue B, then you bind the queue A with a binding key of twice the number (as a string -- binding keys are always strings) of the binding key of the binding to queue B." – ospider Aug 08 '18 at 02:06
  • I'm using the latest version of [stable/rabbitmq-ha](https://github.com/helm/charts/tree/master/stable/rabbitmq-ha) helm chart and it does **NOT** have `x-modulus-hash` leading to this error: `unknown exchange type 'x-modulus-hash'`. If you know when `x-modulus-hash` was discontinued, please answer [this question](https://stackoverflow.com/questions/59947668/latest-rabbitmq-release-with-exchange-type-x-modulus-hash). – mhyousefi Jan 28 '20 at 11:34