Time limit for pausing kafka consumer

Question

I wanted to implement a functionality which requires Kafka queue to be paused and resume, what I want's to know that is there any time limit upto which it can be paused?

Answer 1

Kafka doesn't really have "queues", all messages in a topic are there to be consumed by Consumers. Your Consumers can consume messages in the way they prefer, a Consumer can start consuming messages from the beginning or from any offset they want, they can also stop and resume as they want. When a Consumer consumes messages, it can commit the offsets back to Kafka, if the consumer dies, when it will be back, it will start from the last committed message.

If what you want is to poll a bunch of messages and do something with them for a long period of time, Kafka Consumers have a configuration max.poll.interval.ms that by default is 5 minutes. If you expect to consume a message and to be doing something with it for more than 5 minutes, you should increase that configuration, otherwise the consumer group will think your Consumer has died and will rebalance partitions.