0

I'm using kafka 0.9.0 with pykafka as a url queue in scrapy, and the topic i used have 30 partitions with 4 consumers.

It works properly at first, but when it worked about a couple of hours the kafka seems having a problem with rebalancing the partitons, and i got some infomations like

Unable to acquire partition < pykafka.partition.Partition at 0x7f0b422cbfd0 (id=23) >. Retrying

than it raise two exceptions

pykafka.exceptions.ConsumerStoppedException

pykafka.exceptions.PartitionOwnedError

So I want to know the allocation of kafka partition. i use the kafka-consumer-groups.sh like

./kafka-consumer-groups.sh --zookeeper host:port --group name --describe

which mention on kafka's document, but i got a empty list.

jason
  • 11
  • 4
  • In 0.9.0.0, you have to make the consumer group active before checking its status. – amethystic Apr 25 '17 at 07:47
  • 1
    i'm new to kafka so first i want to know what is meaning of an inactive group. Is the so call inactive means that there is no consumer in that group consume topic recently? And I found this problem is 'normal' in pykafka's balancedconsumer, when i use kafka-python all things work fine, so i think the pykafka has some problem itself @amethystic – jason Apr 27 '17 at 07:35
  • Did you run kafka-consumer-groups after you stop the consumer application? – amethystic Apr 27 '17 at 09:59
  • Please add your code to question – Mohammad Hossein Gerami Mar 11 '18 at 12:36

0 Answers0