Currently, I am using KafkaSource to get the records from multiple event streams. Each event stream is having one partition. I would like to check if rebalancing happens across the Flink app's having same consumer group.
Expectation: One Flink application would process the message while other one won’t get any message as they both are in the same consumer grp.
Actual: Both the app’s are getting the same message, even though the consumer group name is same.
Is this the expected behavior?
Flink's Kafka source does not rely on Kafka consumer groups to consume from Kafka. It is kind of implied in here: https://nightlies.apache.org/flink/flink-docs-stable/docs/connectors/datastream/kafka/#consumer-offset-committing:
Note that Kafka source does NOT rely on committed offsets for fault tolerance. Committing offset is only for exposing the progress of consumer and consuming group for monitoring.
It doesn't explicitly say so, but other than using committed offsets as a starting point (with setStartingOffsets) and to show progress, Flink's Kafka source manages partition assignment at a lower level and does not rely on consumer groups. So it is expected behavior
