0

I'm planning to use Kafkajs https://kafka.js.org/ and implement it in a NodeJs server. I would like to know what is the expected behavior in case I have 2 (or more) instances of the server running, each of them having a consumer which is configured with the same group id and topic?

Does this mean that they might read the same messages? Should I specify a unique consumer group per each server instance ?

I read this - Multiple consumers consuming from same topic but not sure it applies for Kafkajs

David Faizulaev
  • 4,651
  • 21
  • 74
  • 124

1 Answers1

1

It's not possible for a single consumer group to have multiple consumers reading from overlapping partitions in any Kafka library. If your topic only has one partition, only one instance of your application will be consuming it, and if that instance dies, the group will rebalance and the other instance will take over (potentially reading some of the same data, due to the nature of at-least-once delivery, but it's not at the same time as the other instance)

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • I have one more question, in case I have 2 consumers, A - took messages 1 - 100 B - took messages 101 - 200 A failed, B succeeded what happens to the messages 1 - 100? – David Faizulaev Feb 13 '22 at 10:44
  • 1) As answered, that's only possible with multiple partitions since multiple consumers cannot be assigned the same one 2) The answer is the same thing that happened to offsets 1-100 of the second partition – OneCricketeer Feb 13 '22 at 15:16
  • so until they are committed, messages 1-100 in partition 0, are waiting for a consumer to read them? (disregarding the retention time) – David Faizulaev Feb 13 '22 at 18:34
  • 1
    Unless a consumer skips them with a seek method, yes – OneCricketeer Feb 14 '22 at 00:18