Spring Kafka - How to Retry with @KafkaListener

Question

Question from Twitter:

Just trying to find out a simple example with spring-kafka 2.1.7 that works with a KafkaListener and AckMode.MANUAL_IMMEDIATE , to retry last failed message.

https://twitter.com/tolbier/status/1028936942447149056

Answer 1

it's generally better to ask such questions on Stack Overflow (tagged with spring-kafka.

There are two ways:

• Add a RetryTemplate to the listener container factory - the retries will be performed in memory and you can set backoff properties.
• Add a SeekToCurrentErrorHandler which will re-seek the unprocessed records.

Here is an example:

@SpringBootApplication
public class Twitter1Application {

    public static void main(String[] args) {
        SpringApplication.run(Twitter1Application.class, args);
    }

    boolean fail = true;

    @KafkaListener(id = "foo", topics = "twitter1")
    public void listen(String in, Acknowledgment ack) {
        System.out.println(in);
        if (fail) {
            fail = false;
            throw new RuntimeException("failed");
        }
        ack.acknowledge();
    }

    @Bean
    public ConcurrentKafkaListenerContainerFactory<?, ?> kafkaListenerContainerFactory(
            ConcurrentKafkaListenerContainerFactoryConfigurer configurer,
            ConsumerFactory<Object, Object> kafkaConsumerFactory) {
        ConcurrentKafkaListenerContainerFactory<Object, Object> factory = new ConcurrentKafkaListenerContainerFactory<>();
        configurer.configure(factory, kafkaConsumerFactory);
        factory.getContainerProperties().setErrorHandler(new SeekToCurrentErrorHandler());
        // or factory.setRetryTemplate(aRetryTemplate);
        // and factory.setRecoveryCallback(aRecoveryCallback);
        return factory;
    }

    @Bean
    public ApplicationRunner runner(KafkaTemplate<String, String> template) {
        return args -> {
            Thread.sleep(2000);
            template.send("twitter1", "foo");
            template.send("twitter1", "bar");
        };
    }

    @Bean
    public NewTopic topic() {
        return new NewTopic("twitter1", 1, (short) 1);
    }

}

and

spring.kafka.consumer.auto-offset-reset=earliest
spring.kafka.consumer.enable-auto-commit=false

spring.kafka.listener.ack-mode=manual-immediate

logging.level.org.springframework.kafka=debug

and

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>

    <groupId>com.example</groupId>
    <artifactId>twitter1</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <packaging>jar</packaging>

    <name>twitter1</name>
    <description>Demo project for Spring Boot</description>

    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>2.0.4.RELEASE</version>
        <relativePath/> <!-- lookup parent from repository -->
    </parent>

    <properties>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
        <project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
        <java.version>1.8</java.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.kafka</groupId>
            <artifactId>spring-kafka</artifactId>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
            </plugin>
        </plugins>
    </build>


</project>

(Boot 2.0.4 pulls in 2.1.8, which is the current version).

and

foo
2018-08-13 17:36:14.901 ERROR 3945 --- [      foo-0-C-1] essageListenerContainer$ListenerConsumer : Error handler threw an exception

org.springframework.kafka.KafkaException: Seek to current after exception; nested exception is  ...    

2018-08-13 17:36:15.396 DEBUG 3945 --- [      foo-0-C-1] essageListenerContainer$ListenerConsumer : Received: 2 records
foo
2018-08-13 17:36:15.398 DEBUG 3945 --- [      foo-0-C-1] essageListenerContainer$ListenerConsumer : Committing: {twitter1-0=OffsetAndMetadata{offset=5, metadata=''}}
bar
2018-08-13 17:36:15.403 DEBUG 3945 --- [      foo-0-C-1] essageListenerContainer$ListenerConsumer : Committing: {twitter1-0=OffsetAndMetadata{offset=6, metadata=''}}

In the upcoming 2.2 release, the error handler can be configured with a recoverer and standard recoverer is provided to publish the failed record to a dead-letter topic.

Commit here. Docs Here.