2

Kafka rebalancing is designed to redistribute all the partitions of topic to subscriber group alive members so that any of the topic partitions is used by only one consumer at any given moment. So in case if consumers have just subscribed to a topic, everything is clear, but consumer api also provides a way to assign a specific partition to a consumer:

void assign(Collection<TopicPartition> partitions)

Now lets assume we have:

  • a topic with 5 partitions: 1,2,3,4,5
  • a consumer group with 3 members: 1,2,3
  • members 1 and 2 has just subscribed to a topic, member 3 has assigned partition 1 to itself

So how the rebalancing will work in such case ? Will the partition 1 always be assigned to member 3(while it is alive) resulting that other members can read only from other partitions(2,3,4,5) ? Can group coordinator assign also another partitions to member 3 ? What happens to partition 1 in case when member 3 goes down and later comes back ?

Thanks in advance

Suren Aznauryan
  • 984
  • 10
  • 24

1 Answers1

1

Found in kafka docs:

public void assign(Collection<TopicPartition> partitions)

Manual topic assignment through this method does not use the consumer's group management functionality. As such, there will be no rebalance operation triggered when group membership or cluster and topic metadata change. Note that it is not possible to use both manual partition assignment with assign(Collection) and group assignment with subscribe(Collection, ConsumerRebalanceListener).

So all of the consumers in group should use either manual partition assignment or just subscription(getting dynamic assignment of partitions).

Community
  • 1
  • 1
Suren Aznauryan
  • 984
  • 10
  • 24