Is there any possible to develop a bi-directional messaging system using apache kafka ? I need to subscribe for a topic from my consumer as well as I need to send message from my consumer.
Asked
Active
Viewed 4,882 times
6
-
1Use two message topics? – jaco0646 Sep 22 '16 at 14:03
-
@jaco0646 How can I send a message from my consumer to server ? – Prasath Sep 27 '16 at 07:14
-
Put a message on a topic that the server listens to. – jaco0646 Sep 27 '16 at 12:50
-
1So for a bi-directional messaging system I need to create two producers and two consumers. – Prasath Sep 27 '16 at 13:09
-
1Yes. That's correct. – jaco0646 Sep 27 '16 at 13:45
-
Ok thank you for your advice @jaco0646 – Prasath Sep 28 '16 at 06:15
-
Hi Prasath, any luck on finding a way for 2way communication with kafka? – Gaurav Ram Mar 17 '17 at 20:04
-
@GauravRam No luck. I started my design with 2 producers & 2 consumers. – Prasath Mar 20 '17 at 05:51
1 Answers
3
You could do it one of two ways. Either set up a prefix system for the message keys or put content inside of the message that allows the consumer to avoid messages it has produced.
Now as to whether you should design it like this, that depends on your message traffic. If you're not slamming it with events, it might be better to consider something like Thrift as a way to have your message components do bidirectional communication. Where Kafka really excels relative to its complexity is when you need to produce and consume massive volumes of data. That might not be the case for you.
For example, one common use case with Kafka is to hook it up to a service like Storm, Apex or Samza for doing distributed processing of hundreds of GB or even TB of data. If your system has a high throughput requirement, that architecture would be a good one to consider as a starting point with Kafka for handling messages. With Storm, if you need to send messages back for reprocessing, you can always use the Kafka bolt to republish a message into Kafka to ensure it gets completely reprocessed.
Mike Thomsen
- 36,828
- 10
- 60
- 83