When I start consumer with single instance it is show in consumer group but it is not consuming data from topic. After that if I start another consumer and my first consumer start consuming data but latest consumer instance doesn't have any partition assigned to it.
below is the info log when first consumer instance starts.
INFO:kafka.client:Bootstrapping cluster metadata from [(u'kafka-broker1.ap-south-1.staging.internal', 9092, 0)] INFO:kafka.conn:: connecting to 172.31.1.66:9092 INFO:kafka.client:Bootstrap succeeded: found 3 brokers and 19 topics. INFO:kafka.conn:: Closing connection. INFO:kafka.conn:: connecting to 172.31.1.148:9092 INFO:kafka.conn:Broker version identifed as 0.11.0 INFO:kafka.conn:Set configuration api_version=(0, 11, 0) to skip auto check_version requests on startup INFO:kafka.consumer.subscription_state:Subscribing to pattern: /events/ INFO:kafka.conn:: connecting to 172.31.1.70:9092 INFO:kafka.cluster:Group coordinator for datadog is BrokerMetadata(nodeId=1, host=u'kafka-broker1.ap-south-1.staging.internal', port=9092, rack=None) INFO:kafka.coordinator:Discovered coordinator 1 for group datadog INFO:kafka.conn:: connecting to 172.31.1.66:9092 INFO:kafka.coordinator.consumer:Revoking previously assigned partitions set([]) for group datadog INFO:kafka.coordinator:(Re-)joining group datadog INFO:kafka.consumer.subscription_state:Updating subscribed topics to: [u'events'] INFO:kafka.coordinator:Joined group 'datadog' (generation 843) with member_id kafka-python-1.3.5-e3c25fb3-39ea-4550-845f-9b663355b4f5 INFO:kafka.coordinator:Successfully joined group datadog with generation 843 INFO:kafka.consumer.subscription_state:Updated partition assignment: [] INFO:kafka.coordinator.consumer:Setting newly assigned partitions set([]) for group datadog
When I start second instance first instance get partition assigned and second instance have 0 partition assigned and have same info log as first instance before starting second instance.
below is the info log of first instance after second instance started.
INFO:kafka.coordinator.consumer:Setting newly assigned partitions set([]) for group datadog WARNING:kafka.coordinator:Heartbeat failed for group datadog because it is rebalancing WARNING:kafka.coordinator:Heartbeat failed ([Error 27] RebalanceInProgressError); retrying INFO:kafka.coordinator.consumer:Revoking previously assigned partitions set([]) for group datadog INFO:kafka.coordinator:(Re-)joining group datadog INFO:kafka.coordinator:Skipping heartbeat: no auto-assignment or waiting on rebalance INFO:kafka.coordinator:Joined group 'datadog' (generation 843) with member_id kafka-python-1.3.5-ddb66185-c615-4f31-9729-9384131f24c9 INFO:kafka.coordinator:Elected group leader -- performing partition assignments using range INFO:kafka.coordinator:Successfully joined group datadog with generation 843 INFO:kafka.consumer.subscription_state:Updated partition assignment: [TopicPartition(topic=u'events', partition=0), TopicPartition(topic=u'events', partition=1), TopicPartition(topic=u'events', partition=2), TopicPartition(topic=u'events', partition=3), TopicPartition(topic=u'events', partition=4), TopicPartition(topic=u'events', partition=5), TopicPartition(topic=u'events', partition=6), TopicPartition(topic=u'events', partition=7), TopicPartition(topic=u'events', partition=8), TopicPartition(topic=u'events', partition=9)] INFO:kafka.coordinator.consumer:Setting newly assigned partitions set([TopicPartition(topic=u'events', partition=6), TopicPartition(topic=u'events', partition=7), TopicPartition(topic=u'events', partition=8), TopicPartition(topic=u'events', partition=9), TopicPartition(topic=u'events', partition=0), TopicPartition(topic=u'events', partition=1), TopicPartition(topic=u'events', partition=2), TopicPartition(topic=u'events', partition=3), TopicPartition(topic=u'events', partition=4), TopicPartition(topic=u'events', partition=5)]) for group datadog
We always have 1 consumer instance with unassigned partitions on all the cases, last consumer instance always have 0 partition assigned
**Below are the screenshot for the same **
No partitions assigned to first consumer instance
No partitions assigned to last consumer instance
We also suspect clustering of kafka. When we had only 1 kafka node partition assignment to consumer instance and re-balancing was working fine. But after going into multi-node cluster we are facing this issue
