This is the base consumer class I'm using for creating new consumers. It works fine for "enable.auto.commit":True
consumer. But when I create a consumer with enable.auto.commit=False
and any of the (KeyDeserializationError, ValueDeserializationError) exceptions occurs then I need to manually commit that message in except block. As this base class will be used for auto-commit=True as well, so this line self.consumer.commit() is getting called to these types of consumers also.
- By calling commit() for
auto.commit=True
consumers give any issue internally? (it seems fine when I tried locally) - What should be ideal handling for (
KeyDeserializationError
,ValueDeserializationError
) exceptions forauto.commit=False
?
class KafkaConsumer(object):
"""Wrapper over Kafka Consumer"""
def __init__(self, topics: list[str], **kwargs: Any):
config = {
**kwargs,
}
self.consumer = DeserializingConsumer(config)
self.consumer.subscribe(topics=topics)
def consume(self, poll_timeout_secs: float = 1.0):
try:
while True:
try:
msg = self.consumer.poll(timeout=poll_timeout_secs)
except (KeyDeserializationError, ValueDeserializationError) as err:
self.consumer.commit()
continue
if msg is None:
continue
if msg.error():
raise KafkaException(msg.error())
else:
yield msg
except:
self.consumer.close()
# create consumer object auto.commit=True/False
kafka_consumer = KafkaConsumer(topics=topics, **kwargs) # i can pass "enable.auto.commit":False for manual commit mode.
# Actual consuming business logic
for message in kafka_consumer.consume():
try:
event = message.value()
logger.info(f"message {event}")
except Exception as e:
logger.exception(f'Unable to consume data from kafka {e}')
finally:
pass
# kafka_consumer.consumer.commit(message=message) # in case of manual commit consumer mode