Set polling rate for kafka consumers per topic using springboot

Question

Is it possible to set different poll rates for consumers each kafka topics on SpringBoot? I would like to have one topic to be polling with longer intervals (like 5 minutes). I was hoping to use this to implement a retry for failed processing of messages from kafka.

A sample implementation will greatly help.

Refer to this question https://stackoverflow.com/questions/51753883/increase-the-number-of-messages-read-by-a-kafka-consumer-in-a-single-poll — TechGeek, Aug 23 '20 at 06:01
@Arun thanks for the response. But I wanted to change the interval between polling and not change the number of messages retrieved for every poll of the consumer, I updated my question to make it clearer. — itabangay, Aug 24 '20 at 02:23
you can do it, create different consumers for each topic @itabangay — Ryuzaki L, Aug 24 '20 at 02:44
@deadpool can you point me to a sample implementation on how to do this? — itabangay, Aug 24 '20 at 03:35

score 0 · Accepted Answer · answered Aug 24 '20 at 14:12

Since version 2.3, the listener container has a new property idleBetweenPolls. The container will ensure that the time between polls is no larger than max.poll.interval.ms - 5000.

You can set it for specific consumers using a container customizer bean...

@Component
class Customizer {

    public Customizer(ConcurrentKafkaListenerContainerFactory<?, ?> factory, DefaultKafkaProducerFactory<?, ?> pf) {
        factory.setContainerCustomizer(container -> {
            if (container.getContainerProperties().getGroupId().equals("slowGroup")) {
                // or you can use the topic(s)
                container.getContainerProperties().setIdleBetweenPolls(60_000);
            }
        });
    }

}

Set polling rate for kafka consumers per topic using springboot

1 Answers1