0

Let's say I have an Order and a Product.

Whenever product is created I publish PRODUCT_CREATE event and for the order it is ORDER_CREATE

If I use Kafka ( or any message broker ) should I have 1 queue per microservice and process events one-by-one or should I have queue-per-feature?

Lets say I quickly create a product and then order ( almost instantly )

Solution 1: I have 1 queue. I can check if product exists when I try to create order. If it doesn't I throw an error because everything is inconsistent.

Solution 2: I have 2 queues. One for PRODUCT_CREATE and one for ORDER_CREATE. If I try to create an order and product is not there I can retry until product emerges. At some point I can say product is not there and move on.

What is the recommended solution?

I tried Solution 2 with retries. It works, but what if product truly doesnt exist? It seems like a dirty approach.

And Solution 1 is simple, but I cant do work in parallel at all.

Guru Stron
  • 102,774
  • 10
  • 95
  • 132
0xDjole
  • 25
  • 3

1 Answers1

0

1.If business is coupled tightly, solution 1 is recommended.

Don't worry about parallelling, kafka provides partitons to increase throughput. You can send PRODUCT_CREATE and ORDER_CREATE event with same key (productId), and mutilple consumers can consume the events orderly.

Let's say your business have five kinds of event, it's impossible to have five queues and recover the order on the consumer side. So, the key is how tightly your business is coupled.

2.Another way to improve solution 2 is kafka transaction, it ensures PRODUCT_CREATE and ORDER_CREATE event are sent to kafka brokers in an atomic way.

producer.initTransactions();
producer.beginTransaction();
producer.send("outputTopic", "message1");
producer.send("outputTopic", "message2");
producer.commitTransaction();
Mathew
  • 456
  • 3
  • 5
  • Thank you for a response. But in terms of parallel processing, Im concerned more on consumer side doing processing all events one-by-one as they are grouped. I know kafka can handle it, but can my server process it.. – 0xDjole May 11 '23 at 13:15
  • You can do parallel processing in your server, but actually you are doing things what kafka can do. – Mathew May 11 '23 at 13:56
  • 2
    _"You can send PRODUCT_CREATE and ORDER_CREATE event with same key (productId), and mutilple consumers can consume the events orderly."_ - what if order has product ids from different partitions? – Guru Stron May 11 '23 at 18:18
  • @GuruStron We need to know more about of producer business and consumer business. In your case, the producer can send PRODUCT_INFO and ORDER_CREATE event to kafka. – Mathew May 12 '23 at 00:16
  • Sure, two events could be sent, but that doesn't answer any guarantee about ordering. Plus, products and orders are probably going to be different topics. Therefore same partition number, or record key, means nothing – OneCricketeer May 13 '23 at 13:13