1

This is the base consumer class I'm using for creating new consumers. It works fine for "enable.auto.commit":True consumer. But when I create a consumer with enable.auto.commit=False and any of the (KeyDeserializationError, ValueDeserializationError) exceptions occurs then I need to manually commit that message in except block. As this base class will be used for auto-commit=True as well, so this line self.consumer.commit() is getting called to these types of consumers also.

  1. By calling commit() for auto.commit=True consumers give any issue internally? (it seems fine when I tried locally)
  2. What should be ideal handling for (KeyDeserializationError, ValueDeserializationError) exceptions for auto.commit=False?
class KafkaConsumer(object):
    """Wrapper over Kafka Consumer"""
    def __init__(self, topics: list[str], **kwargs: Any):
        config = {
            **kwargs,
        }
        self.consumer = DeserializingConsumer(config)
        self.consumer.subscribe(topics=topics)

    def consume(self, poll_timeout_secs: float = 1.0):
        try:
            while True:
                try:
                    msg = self.consumer.poll(timeout=poll_timeout_secs)
                except (KeyDeserializationError, ValueDeserializationError) as err:
                    self.consumer.commit()
                    continue
                if msg is None:
                    continue
                if msg.error():
                    raise KafkaException(msg.error())
                else:
                    yield msg
        except:
            self.consumer.close()

# create consumer object auto.commit=True/False

kafka_consumer = KafkaConsumer(topics=topics, **kwargs) # i can pass "enable.auto.commit":False for manual commit mode.

# Actual consuming business logic

for message in kafka_consumer.consume():
    try:
        event = message.value()
        logger.info(f"message {event}")
    except Exception as e:
        logger.exception(f'Unable to consume data from kafka {e}')
finally:
pass
# kafka_consumer.consumer.commit(message=message) # in case of manual commit consumer mode
OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
Akash Pagar
  • 637
  • 8
  • 22

0 Answers0