How to reduce Kafka lag? and How to create multiple consumers for same topic?

Question

def getKafkaConsumer(topicName, kafkaIP, kafkaPort, kafkaConsumerGroup): try: bootstrap = [str(kafkaIP + ':' + kafkaPort)] consumer = KafkaConsumer(topicName, bootstrap_servers = bootstrap, group_id = kafkaConsumerGroup, session_timeout_ms = 30000, api_version = (0,10)) return consumer except: import traceback print('Kafka Error: Check Kafka Consumer Settings!!')

In my application, producer is producing approx. 1000 messages per second but consumption rate is very slow. In above python code of consumer, please suggest how can I tune kafka parameters to increase consumption. And also suggest how can I create multiple consumers for same Topic.

you should format your code to make it easier to read. – Frank C. Aug 20 '22 at 14:41 — Frank C., Aug 20 '22 at 14:41

score 0 · Answer 1 · answered Aug 20 '22 at 17:30

0

The max number of consumers you can spawn per Kafka topic is equal to the number of partitions in a kafka topic.

So increase the number of partitions and launch multiple consumers.

answered Aug 20 '22 at 17:30

floating_hammer

409
3
10

Thanks for your response, Could you please elaborate on launching multiple consumers, I mean how to create multiple consumers in kafka-python? – Rishikesh mahajan Aug 21 '22 at 08:26
Launch them as separate applications but connecting to the same consumer group – floating_hammer Aug 21 '22 at 21:22

How to reduce Kafka lag? and How to create multiple consumers for same topic?

1 Answers1