how to improve python-kafka consume throughout

Asked Jan 04 '18 at 14:24

Active Jan 05 '18 at 03:30

Viewed 132 times

I test my python-kafka script, the kafka-producer produced 10000/s message, but I consumed about 1000/s, so how to improve my consumer throughout?

while True:
    topic_to_records = consumer.poll(100)
    if len(topic_to_records.values()) == 0:
        time.sleep(0.05)
    for records in topic_to_records.values():
        print 'consume %s messages' % len(records)
        for msg in records:
            print (time.strftime('%Y-%m-%d %H:%M:%S', time.localtime()), '*'+msg.value.split('\t')[0])

edited Jan 05 '18 at 03:30

asked Jan 04 '18 at 14:24

lizx

Could you share at least how to you are consuming the data. How many partitions we are talking about. How many consumers processes. – dbustosp Jan 04 '18 at 21:19
only one consumer process, about 30 partitions, – lizx Jan 05 '18 at 03:30

how to improve python-kafka consume throughout

0 Answers0