13

I am using Spring Kafka consumer which fetches messages from a topic and persist them into a db. If a failure condition is met , say for instance the db is unavailable , does kafka consumer library provide mechanism to retry ? If it does , is there a way to set different retry intervals such as 1st retry should be done after 5 mins , 2nd after 30 mins , 3rd after 1 hr etc.

Punter Vicky
  • 15,954
  • 56
  • 188
  • 315

1 Answers1

15

Spring Kafka is shipped with the RetryingMessageListenerAdapter and RetryingAcknowledgingMessageListenerAdapter. If you use @KafkaListener, you can supply AbstractKafkaListenerContainerFactory with the RetryTemplate. And the last one can be injected with any custom RetryPolicy and BackOffPolicy from the Spring Retry project:

https://docs.spring.io/spring-kafka/reference/html/#retrying-deliveries

Also bear in mind that since version 2.0, there is transaction support in Spring Kafka, based on such one in the Apache Kafka 0.11.x.x:

https://docs.spring.io/spring-kafka/reference/html/#transactions

akokskis
  • 1,486
  • 15
  • 32
Artem Bilan
  • 113,505
  • 11
  • 91
  • 118
  • 1
    Thanks Artem , when I have 10 offsets say for instance in a partition. If message at offset 5 is failed to be inserted into DB , would this be tried based on the retry config that I set while offsets 6 through 10 continue to executed as part of normal process? or would consumer wait till 5 is completed successfully? I assume the message will not be committed for offset 5 until retry tries are exhausted – Punter Vicky Oct 02 '17 at 19:44
  • 2
    That's correct. Records in the same partition are processed sequentially and, therefore, `6` won't be processed until `5` is successful or exhausted in retry. – Artem Bilan Oct 02 '17 at 19:59
  • Thanks , one last question. Can I enable retry only for specific types of error? – Punter Vicky Oct 02 '17 at 20:05
  • Yes, you can. That's `RetryPolicy` responsibility. See its JavaDocs and implementations. – Artem Bilan Oct 02 '17 at 20:06