So, we have a producer which was running fine for a few months with no errors but has suddenly started erroring out with
'BufferError: Local: Queue full'
I have had this issue initially, and then by going through documentation and S.O posts , understood that we need to call poll, right after produce, which among many places has also been explained nicely here.
for clientvaluescore in clientvaluescores:
kafka_producer.produce(topic=kafkaconfig['topic_name'],value=clientvaluescore,on_delivery=kafka_delivery_report)
kafka_producer.poll(0)
kafka_producer.flush()
And therefore, I had added poll, which made the error disappear, but after 3 months , I again see the same error. This time, I came across this, and so I added the exception handling as well, along with lingering.ms. This time, however, though I do not get BufferError anymore, I do not see messages being published to the topic. That indicates that publishing has been silently failing.
for churnscore in churnscores:
while True:
try:
kafka_producer.produce(topic=kafkaconfig['topic_name'],value=churnscore,on_delivery=kafka_delivery_report)
kafka_producer.poll(0)
break
except BufferError:
kafka_producer.poll(1)
kafka_producer.flush()
What else should I be doing?
