-1

I have a scenario where system A is going to get information from system B.

Options/Scenarios

  • B will send message to a topic for A to consume along with other consumers of the same topic Pros - for B things will be simple Cons - A has to consume all messages where many messages are not relevant to A

  • B will send message to a topic meant for only A to consume Pros - A will only get the messages that are relevant to A Cons - B has to send to a different topic meant only for A's use case

  • B will a rest API hosted by A Pros - A does not have to worry about message consumption Cons - B has to call API for one consumer A and this is not a scalable approach.

To give more information, in my case the messages relevant to A is very small in number compared to all messages that B sends to A

If the number of messages that A is interested in is very very less (say couple of 100s in x days) vs number of total messages published in the topic (few millions in x days) - then does it justify A to read all these from the topic as in option 1.

C should not sound good from the producer (B) perspective as B can not call API for every new consumer with similar need.

While I am working to find out better solutions (if any) would like to know other thoughts.

Updated below

Option 4 B will send message to a topic for A to consume along with other consumers of the same topic but with key that can be used as filtration/rejection by A.

Option 4 sounds a better one and will explore more.

ctimus
  • 87
  • 9
  • I can't see any other option than 1) be viable in the long run; what if the information A is interested in changes? You'll have to adjust B for the other cases. Option 2 might be applicable, but that's really dependent on the overall architecture of your system - if there are too many messages on a topic not meant for a consumer, that begins to become a smell. – daniu Dec 10 '19 at 09:34

1 Answers1

1

Option 4 is a better idea as in that case A will only read messages relevant to it. So while configuring your consumer for A you can only assign the partition which holds the partition key for A.

Fatema Khuzaima Sagar
  • 359
  • 4
  • 17