I have a Kafka topic gbtopic with 12 partitions and a consumer group gbconsumers with three consumers C1, C2, C3 subscribing to this topic. When all consumers are started, each consumer is allocated with 4 partitions. Then I stopped consumer C3. Now partitions are rebalanced with 6 partitions each to C1 and C2. Now I started consumer C3, and partitions are rebalanced and allocated to C1, C2, C3. My expectation is to get the same set of partitions back with C1, C2, C3 as was assigned before stopping C3.
I am using StickyAssignor for partition.assignment.strategy.
I am performing the operations in the following order:
Start C1 Start C2 Start C3 Stop C3 Restart C3Following is my client program:
public class ConsumerApp {
public static void main(String[] args) {
Scanner scanIn = new Scanner(System.in);
System.out.print("Enter Consumer Name :: ");
String consumerId = scanIn.nextLine().trim();
scanIn.close();
Properties props = new Properties();
props.put(ConsumerConfig.BOOTSTRAP_SERVERS_CONFIG, "localhost:9092");
props.put(ConsumerConfig.GROUP_ID_CONFIG, "gbconsumers");
props.put(ConsumerConfig.ENABLE_AUTO_COMMIT_CONFIG, "true");
props.put(ConsumerConfig.AUTO_COMMIT_INTERVAL_MS_CONFIG, "1000");
props.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
props.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, "30000");
props.put(ConsumerConfig.KEY_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringDeserializer");
props.put(ConsumerConfig.VALUE_DESERIALIZER_CLASS_CONFIG, "org.apache.kafka.common.serialization.StringDeserializer");
props.put(ConsumerConfig.PARTITION_ASSIGNMENT_STRATEGY_CONFIG, StickyAssignor.class.getName());
props.put(ConsumerConfig.CLIENT_ID_CONFIG, consumerId);
KafkaConsumer<String, String> kafkaConsumer = new KafkaConsumer<>(props);
ConsumerRebalanceListener listener = new ConsumerRebalanceListener() {
@Override
public void onPartitionsRevoked(Collection<TopicPartition> collection) {
System.out.println("-------------------------------------------------------------------------------------");
System.out.println(consumerId + " :: onPartitionsRevoked :: " + collection);
}
@Override
public void onPartitionsAssigned(Collection<TopicPartition> collection) {
System.out.println(consumerId + " :: onPartitionsAssigned :: " + collection + "\n");
}
};
kafkaConsumer.subscribe(Collections.singletonList("gbtopic"), listener);
while (true) {
ConsumerRecords<String, String> records = kafkaConsumer.poll((100));
for (ConsumerRecord<String, String> record : records) {
performSomeOperation(record);
}
}
}
private static void performSomeOperation(ConsumerRecord<String, String> record) {
// operations
}
}
Output when all three consumers running, before stopping C3
C1 :: onPartitionsAssigned :: [gbtopic-10, gbtopic-11, gbtopic-8, gbtopic-9]
C2 :: onPartitionsAssigned :: [gbtopic-2, gbtopic-3, gbtopic-4, gbtopic-5]
C3 :: onPartitionsAssigned :: [gbtopic-0, gbtopic-1, gbtopic-6, gbtopic-7]
Output after stopping and starting C3:
C1 :: onPartitionsAssigned :: [gbtopic-0, gbtopic-10, gbtopic-11, gbtopic-6]
C2 :: onPartitionsAssigned :: [gbtopic-4, gbtopic-5, gbtopic-1, gbtopic-7]
C3 :: onPartitionsAssigned :: [gbtopic-2, gbtopic-3, gbtopic-8, gbtopic-9]
I expect the partitions assigned to each consumer after stop/restart C3 to be the same as they were before stop/restart. Am I missing any property or is there any other way to achieve this?
