0

Free Consumers as discussed in this answer are consumers without a group.id. They do not subscribe to the topic but topic partitions are assigned to them.

Can someone let me know how the free consumers behave when they are assigned the same topic partition (auto.offset.reset set to earliest). Are all the free consumers expected to get all the messages till they are restarted?

  • Kafka Client Version - 3.4.0
tuk
  • 5,941
  • 14
  • 79
  • 162

1 Answers1

0

It's not possible for multiple consumers to be assigned to the same partition in a group.

Only when a group rebalances, for any reason, will partitions be reassigned and may be consumed by previously idle consumers.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
  • But I am observing the behaviour different. I have two consumers with no `group.id` and `client.id` set and if I use `assign()` to assign same topic partitions (as described in the linked answer in the question) then I am observing both the consumers are receiving all the messages from the topic. The topic I used for my testing has only 1 partition and the same is **assigned** to both the consumers. – tuk May 03 '23 at 18:25
  • 1
    I see. Yes. Assigned consumers do not use consumer groups. Therefore, you must manually store your offsets somewhere external from Kafka since they wouldn't be stored in consumer offsets topic. I misunderstood the title of your post that said subscribed... So, then I'm not sure about auto.offset.reset ; each consumer manages its own commits and storage. I personally find using assign API is a Kafka anti pattern – OneCricketeer May 04 '23 at 02:22