0

I have a set of kafka-python consumers that consume from different kafka topics continuously and parallely.

My question is how to kick off the consumers in parallel using single python script? And what is the best way to manage(start/stop/monitor) these consumers.

if I write ex: run.py 

import consumer1, consumer2, consumer3

consumer1.start()
consumer2.start()
consumer3.start()

It just hangs on consumer1.start() as the script does not return any value and keeps running.

yoga
  • 1

1 Answers1

1

You can have different threads for each consumer to consume messages in parallel. For example you can have:

consumer1_thread = threading.Thread(target=consumer1.start, args=())
consumer2_thread = threading.Thread(target=consumer2.start, args=())
consumer3_thread = threading.Thread(target=consumer3.start, args=())
consumer1_thread.start()
consumer2_thread.start()
consumer3_thread.start()

You, can see the logs of each thread in parallel and write some logic to stop individual thread if required.

Mayank Gupta
  • 65
  • 7