0

Hypothetical (but simpler) scenario:

  • I have many orders in my system.
  • I have external triggers that affect those orders (e.g. webhooks). They may occur in parallel, and are handled by different instances in my cluster.
  • In the scope of a single order, I would like to make sure that those events are processed in sequential order to avoid race conditions, version conflicts etc.
  • Events for different orders can (and should) be processed in parallel

I'm currently toying with the idea of leveraging RabbitMQ with a setup similar to this:

  • use a queue for each order (create on the fly)
  • if an event occurs, put it in that queue

Those queues would be short-lived, so I wouldn't end up with millions of them, but it should scale anyway (let's say lower one-digit thousands if the project grows substantially). Question is whether that's an absolute anti-pattern as far as RabbitMQ (or similar) systems goes, or if there's better solutions to ensure sequential execution anyway.

Thanks!

Philipp Sumi
  • 917
  • 8
  • 20

1 Answers1

1

In my opinion creating ephemeral queues might not be a great idea as there will considerable overhead of creating and delete queues. The focus should be on message consumption. I can think of following solutions:

  • You can limit the number of queues by building publishing strategy like all orders with orderId divisible by 2 goes to queue-1, divisible by 3 goes to queue-2 and so forth. That will give you parallel throughput as well as finite number queues but there is some additional publisher logic you have to handle

  • The same logic can be transferred to consumer side by using single pub-sub style queue and then onus lies on consumer to filter unwanted orderIds

  • If you are happy to explore other technologies, you can look into Kafka as well where you can use orderId as partitionKey and use multiple partitions to gain parallel throughput.

Sukhmeet Sethi
  • 616
  • 5
  • 14
  • This is good input, thanks! The fixed # of queues is an option I considered since it would still scale by means of configuration, but I would still come with a penalty. I might be able to further reduce that with a scheduler on the consumer side though (triggers parallel processing for the next message if the previous one(s) weren't for the same order, otherwise wait). I currently don't know much about Kafka, but I'll read up on that. Thank you! – Philipp Sumi Dec 14 '21 at 14:38