How to know when consumer.poll() is called while using @KafkaListener annotation?

Question

I understand that I cannot control when to poll if I use @KafkaListener, and I read from this answer that

the next poll() is performed after the last message from the previous poll has been processed by the listener.

So I'm wondering how to know when each poll() is executed? Or equivalently, how long does it take to process all messages received in each poll() call?

I am asking because my program got "Offset commit failed ... The request timed out" exceptions, and I would like to tune my consumer config, i.e. max.poll.interval.ms and max.poll.records, but I need to know the current performance first.

Here is part of my @KafkaListener method if it helps:

@KafkaListener(id = "dataListener", topics ="${spring.kafka.topic}", containerFactory = "kafkaListenerContainerFactory")
public void listen(@Payload(required = false)  ConsumerRecord payload, @Header(KafkaHeaders.RECEIVED_PARTITION_ID) String partition,
                        @Header(KafkaHeaders.OFFSET)Long offset, @Header(KafkaHeaders.RECEIVED_MESSAGE_KEY)String messageKey){
    // processing messages
}

@Gary Russell Can you kindly shed some light? Thanks a lot! – yifei Jul 03 '20 at 14:45 — yifei, Jul 03 '20 at 14:45

score 2 · Accepted Answer · answered Jul 06 '20 at 14:49

2

You can see polling activity by turning on DEBUG logging.

this.logger.debug(() -> "Received: " + records.count() + " records");

logged after each poll().

answered Jul 06 '20 at 14:49

Gary Russell

166,535
14
146
179

Thanks Gary! With this line I can now locate the [source code](https://github.com/spring-projects/spring-kafka/blob/775428b9a8447f50856d6cee5a335c638a05c619/spring-kafka/src/main/java/org/springframework/kafka/listener/KafkaMessageListenerContainer.java), and for those who use `log4j` as i do, you can set `log4j.logger.org.springframework.kafka.listener=debug` but keep `log4j.logger.org.springframework.kafka.listener.adapter=info`, other wise it will log every raw message received. – yifei Jul 08 '20 at 03:31

How to know when consumer.poll() is called while using @KafkaListener annotation?

1 Answers1