0

I'm trying to write a kafka consumer in python using Confluent Kafka. I can get all new messages but I don't get any old messages if I kill and restart my consumer

def confluent_kafka_consumer(app):
    with app.app_context():
        import config
    app.logger.info('Running Confluent Kafka consumer')

    consumer_config = {
        'bootstrap.servers': F'{config.Config.KAFKA_BROKER_URL}:{config.Config.KAFKA_BROKER_PORT}',
        'group.id': 'myGroupId',
        'auto.offset.reset': 'earliest',
        'enable.auto.commit': 'false',
        'max.poll.interval.ms': '86400000'
    }

    try:

        consumer = Consumer(consumer_config)
        consumer.subscribe(['updates'])
        
        while True:

            # read single message at a time
            msg = consumer.poll(0)

            if msg is None:
                gevent.sleep(config.DevelopmentConfig.KAFKA_CONSUMER_THREAD_SLEEP_TIME)
                continue
            if msg.error():
                print("Error reading message : {}".format(msg.error()))
                continue
            # You can parse message and save to data base here
            callstr = msg.value().decode('utf-8')
            print(callstr)

    except Exception as ex:
        print("Kafka Exception : {}", ex)

    finally:
        print("closing consumer")
        consumer.close()

I tried setting groupId to something different, restarting the producer, zookeper...

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
user14530855
  • 1
  • 1

1 Answers1

0

Did you get existing data the first time you ran the code? If so, that's how consumer groups work; they maintain the position across restarts.

You will either need a new group.id, or you can externally use kafka-consumer-groups CLI command (you'll need to download Kafka) to reset the offsets.

Or, producer code simply may be losing data. Do you see the data you're expecting using kafka-console-consumer, for example? Restarts of anything won't help with that.

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245