0

I have deployed Kafka and python consumers on separate servers. I am using GCP VM instances for deployment.

Python scripts run in a loop and checks with kafka if it has a job for it. If it finds a job, it starts the operation otherwise it keeps checking with Kafka.

Scripts works just fine when entries are made in kafka on a regular basis, but exits when no entry is added to kafka queue for a certain amount of time. Thus when I push something into the kafka queue the python script cannot detect it because it has already exited.

Below is the code running on the python to keep checking the kafka for new jobs:

    consumer = KafkaConsumer(bootstrap_servers=['ip-address'], group_id='grp1',
                              value_deserializer=lambda m: json.loads(m.decode('utf8')),
                              partition_assignment_strategy=[RoundRobinPartitionAssignor])
    listener = Listener(consumer1)
    consumer1.subscribe(['topic'], listener=listener)
    while True:
        messages = consumer1.poll(100, max_records=1)
        for tp, msg1 in messages.items():
            # Check partition(s)
            print(tp)
            print(msg1[0])
            print("consumer1")
            print(tp.partition)
            print(msg1[0].value)
            listener.add_offset(tp.topic, tp.partition, msg1[0].offset)
            main() # does some work
            print("done")

Can someone suggest what the problem is?

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Mayank
  • 1,364
  • 1
  • 15
  • 29
  • Is there a traceback when the script exits? Have you tried using cloud functions / Dataflow instead leaving a script running with a while loop? – OneCricketeer Feb 11 '20 at 13:48
  • @cricket_007 There is no traceback before the script exits. But I haven't heard of cloud functions or Dataflow. I will research on those. Thanks! – Mayank Feb 11 '20 at 16:27

0 Answers0