Spring boot Kafka polling until queue is empty and sleep

Question

I am using spring boot with apache kafka. I have a dead letter queue that I want to poll from partition y every x hours, poll until there are no messages left and go back to sleep until the next interval. Is there an option to do that?

Shawrup · Answer 1 · 2020-11-02T18:13:11.423

I faced a similar problem for one of my projects. You have two ways to listen to Kafta topic in spring, one with @KafkaListener and other with KafkaConsumer (official kafka client library).

Edit: As Gary Russel pointed out, If you use @KafkaListener, you can use idleEventInterval and KafkaListenerEndpointRegistry to shutdown and restart @KafkaListener.

But I turned to the other option, using KafkaConsumer of Kafka Client Library. I used Spring @Scheduled for scheduling the polling. I created the consumer and poll for a certain amount of time in a loop, if no new data is found, I close the loop. Spring will start the consumer at the next interval.

My code nearly looked like this.

    // you can add your custom scheduler cron here.
    @Scheduled(fixedDelay = 5*60*1000)
    public void consume(){
        Properties props = new Properties();
        // provide your kafka consumer properties
        props.setProperty("bootstrap.servers", "localhost:9092");
        props.setProperty("group.id", "test");
        props.setProperty("enable.auto.commit", "true");
        props.setProperty("auto.commit.interval.ms", "1000");
        props.setProperty("key.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");
        props.setProperty("value.deserializer", "org.apache.kafka.common.serialization.StringDeserializer");

        KafkaConsumer<String, String> consumer = new KafkaConsumer<>(props);
        // add your topics
        consumer.subscribe(Arrays.asList("foo", "bar"));
        boolean isRunning = true;
        while (isRunning) {
            // kafka consumer will poll for 100 ms, then it will stop polling for this loop.
            ConsumerRecords<String, String> records = consumer.poll(Duration.ofMillis(100));

            if(records.isEmpty()){
                // stop loop is no records are found.
                isRunning = false;
            }

            for (ConsumerRecord<String, String> record : records) {
                System.out.printf("offset = %d, key = %s, value = %s%n", record.offset(), record.key(), record.value());
                // add your business logic here
            }
        }
    }

Add @EnableScheduling to use @Scheduled annotation.

Use the idle event and endpoint registry - see my answer. – Gary Russell Nov 02 '20 at 15:09 — Gary Russell, Nov 02 '20 at 15:09

score 0 · Answer 2 · answered Nov 02 '20 at 15:08

You can configure the listener container with an idleEventInterval; you can then add an event listener to listen for the idle event and stop() the container using the KafkaListenerEndpointRegistry.

See https://docs.spring.io/spring-kafka/docs/2.6.2/reference/html/#events

See https://docs.spring.io/spring-kafka/docs/2.6.2/reference/html/#kafkalistener-lifecycle

Spring boot Kafka polling until queue is empty and sleep

2 Answers2