6

I am new to Kafka and the scenario that i am trying to achieve using Apache Kafka is:

I have a 3 producers who are sending instructions to a particular topic in Kafka. I have around 35 consumers who picks up one instruction at a time and performs the task mentioned in the instruction set. Once completed it will post the updates to an another topic in Kafka.

Requirements.

  1. An instruction read by 1 consumer should not be given to another for processing
  2. Message Priority * - Not a mandatory requirement
  3. Only 1 instruction should be picked up at a time for processing.

I know that this can be achieved using JMS. But my organisation already has a Kafka setup ready as part of Cloud-era HDFS setup. So the using Kafka to achieve this is most preferred.

I have googled around to find a solution for this. But i was not able to find a proper answer to my 1st and most important requirement.

Any suggestions? And thanks in advance.!

tpuli
  • 405
  • 1
  • 5
  • 15
  • Have a look here: http://stackoverflow.com/documentation/apache-kafka/5449/consumer-groups-and-offset-management – Matthias J. Sax Nov 21 '16 at 21:50
  • Furthermore, you can configure each consumer to receive a single message per call via parameter "max.poll.records=1". See http://kafka.apache.org/documentation#newconsumerconfigs – Matthias J. Sax Nov 21 '16 at 21:53

1 Answers1

9

All you have to do is to put the consumers in the same consumer group by assigning the same group.id to them. Only one instance of a consumer in the same group will get a message.

Refer to documentation at https://kafka.apache.org/documentation#intro_consumers

Lundahl
  • 6,434
  • 1
  • 35
  • 37