0

I am facing issue in kafka consumer for execption handling. My flow: 1) subscribe to kafka consumer 2) poll assign to a buffer map 3) commit (Since the process takes more than session time out duration commiting the consumer before processing) 4) process the buffer map load in database 5)If any exception in processing step 4 seek to initial offset number saved in poll step 2

There are two issues, the seek is not working from exception, When I start the consumer again with seek option, I am creating one consumer seeking it to previous successful offset , I confirmed the seek is moving correctly. Then I created another kafka consumer to process the data but the seek operation is not considered, it is picking from committed offset.

sra1
  • 31
  • 2
  • 5
  • org.apache.kafka.clients.consumer.CommitFailedException: Commit cannot be completed since the group has already rebalanced and assigned the partitions to another member. This means that the time between subsequent calls to poll() was longer than the configured max.poll.interval.ms, which typically implies that the poll loop is spending too much time message processing. You can address this either by increasing the session timeout or by reducing the maximum size of batches returned in poll() with max.poll.records. – sra1 May 30 '18 at 04:31
  • example: kafka processed offset 100. code failed after processing 50 records, the committed offset is 150 I want to seek to 100 and reprocess, I am using a method for seeking consumer which will only seek the offset and I am closing it. Then I start the actual consumer and poll the topic. when seeking the offset I could see the offset is changed from 150 to 100 but when I start the consumer again to poll I see the offset as 150. The seek process I am doing is not helping. – sra1 May 30 '18 at 04:34

0 Answers0