I have a Kafka cluster of 3 separate physical nodes, with each hosting 1 zk and 1 broker each. We are using Kafka 0.10.1.1. The Topics are created on ZK ensemble and the brokers connect to the ZK ensemble. My Producer and Consumer is a Spring Boot app (on version 1.5.13 RELEASE). We have a few topics with Single partition (i.e. - Partition 0), min.insync.replicas=2 and replication_factor=3. So far when we check through --describe topics, we find the all the Replicas are in-sync. Reg Consumer Groups, we have seen that only when a consumer groups is active, it shows up through the CLI and also on Kafka-Manager. Till here it's fine.
We have added individual zk:2181 as individual cluster in Kafka-manager for monitoring. We are seeing even when the consumer groups are not active, they are showing up in some of the clusters(i.e. on zookeepers) with different lag values. The node on on which the lag shows for certain topics are not even leader for the same. When we cross check using CLI, the consumer group don't even show up except KMOffsetCache-<<instance-name>>. kafka-consumer-groups.sh --bootstrap-server ...
It appears that either the zk(s) or consumer_offsets are not properly synced up. I'd like to understand why? and how to rectify the same?
