TL;DR: Is there a limitation on re-producing to a topic as part of the callback of a previous production? Consumer hangs with
Application maximum poll interval (300000ms) exceeded by 72ms (adjust max.poll.interval.ms for long-running message processing): leaving group
I get an error on a consumer-producer process.
Using python 3.9, confluent_kafka
library, Producer
/Consumer
classes.
Process uses Consumer
to pull data from topic.
It processes the events and uses the Producer
to produce to a separate topic.
If there is a failure on production because the topic does not exist, the callback uses the AdminClient
to create it (I have a dependency on the type of message to produce to a certain topic, creating a new topic if the type is coming first time).
The creation of the topic, and the production to the topic happens on the callback function.
When the topic is already created, the application works normally. However when the topic is not created:
2023-05-17T13:31:07.976+02:00 Attempted to produce message to non-existent topic: xxx
2023-05-17T13:31:07.976+02:00 Creating new topic xxx and reproducing message
2023-05-17T13:31:08.747+02:00 Topic "xxx" created
2023-05-17T13:31:09.901+02:00 Produced message on topic xxx
2023-05-17T13:35:35.285+02:00 %4|1684323335.285|MAXPOLL|process#consumer-1| [thrd:main]: Application maximum poll interval (300000ms) exceeded by 72ms (adjust max.poll.interval.ms for long-running message processing): leaving group
In this situation, the default max.poll.interval.ms
looks good enough, especially seeing that the topic is created and the message gets produced quick.
It seems like since the production of the message, until the timeout happens, there are 4'20'' where the process goes idle, and so it never comes back?
Consumer commits happen after the production is finished (to do at-least-once
)
Could it be because of an accumulation/conflict of callbacks? Any other idea?