How do you exit spring boot application programmatically when retries are exhausted, to prevent kafka offset commit

Question

I want to exit a spring boot application programmatically when retries are exhausted in order to never miss processing any events. Neither System.exit() nor SpringBootApplication.exit() will terminate the application.

The application is using a @KafkaListener function that saves events to database and has a SeekToCurrentErrorHandler with a customer recoverer. The recoverer is where i have tried to exit the application to prevent the kafka offset from being committed.

The application also has a @Scheduled function, maybe that what is preventing termination??

Googling this leaves me empty. I am not even sure this is the correct way to do this, eg if terminating here will actually prevent offset commit. Here is the code i've tried for configuring recoverer (in kotlin):

    fun kafkaListenerContainerFactory(
        configurer: ConcurrentKafkaListenerContainerFactoryConfigurer,
        kafkaConsumerFactory: ConsumerFactory<Any?, Any?>?
    ): ConcurrentKafkaListenerContainerFactory<*, *>? {
        val factory = ConcurrentKafkaListenerContainerFactory<Any, Any>()
        configurer.configure(factory, kafkaConsumerFactory)
        factory.setErrorHandler(
            SeekToCurrentErrorHandler(
                { _, ex ->
                    //System.exit(-1)
                    SpringApplication.exit(appContext, { -1 })
                })
        )
        return factory
    } 

Answer 1

Boot registers a shutdown hook so that the application context is closed whenever the JVM exits (e.g. with System.exit).

Shutting down the application context includes stopping components such as the listener container; so stopping it there will cause a deadlock for a few seconds.

You should use logic such as that in the ContainerStoppingErrorHandler instead; and throw an exception after stopping the container.

EDIT

Here is an example:

@SpringBootApplication
public class So67076203Application {


    private static final Logger log = LoggerFactory.getLogger(So67076203Application.class);


    public static void main(String[] args) {
        SpringApplication.run(So67076203Application.class, args);
    }

    @Bean
    public NewTopic topic() {
        return TopicBuilder.name("so67076203").partitions(1).replicas(1).build();
    }

    @KafkaListener(id = "so67076203", topics = "so67076203")
    public void listen(String in) {
        System.out.println(in);
        throw new RuntimeException("x");
    }

    @Bean
    ErrorHandler stceh(ConsumerRecordRecoverer recoverer) {
        return new SeekToCurrentErrorHandler(recoverer, new FixedBackOff(2000L, 2L));
    }

    @Bean
    ConsumerRecordRecoverer recoverer(KafkaListenerEndpointRegistry registry) {
        return (cr, ex) -> {
            log.error("Stopping container due to error " + ex.getMessage());
            try {
                stopper().handle(ex, null, null, registry.getListenerContainer("so67076203"));
            }
            catch (KafkaException e) {
                log.error("shutting down the JVM " + e.getMessage());
                new SimpleAsyncTaskExecutor().execute(() -> System.exit(-1));
                throw e;
            }
        };
    }

    // can't be a bean because then Boot won't auto configure the stceh
    private ContainerStoppingErrorHandler stopper() {
        return new ContainerStoppingErrorHandler();
    }

}

@Component
class Customizer {

    Customizer(ConcurrentKafkaListenerContainerFactory<?, ?> factory) {
        factory.getContainerProperties().setCommitLogLevel(Level.INFO);
        factory.getContainerProperties().setStopImmediate(true);
    }

}

EDIT

With versions 2.8 and later, the SeekToCurrentErrorHandler is replaced by the DefaultErrorHandler.

Answer 2

Maybe not the exact same case, but hopefully still useful for someone: I wanted my Spring Boot app to shutdown whenever a Kafka consumer stopped (typically related to authentication/authorization problems). Spring-internal events can be used for that. I'm not claiming that this is the best or most correct solution, but after a couple of other attempts, this was what worked for me:

import org.springframework.context.ApplicationContextAware;
import org.springframework.context.ApplicationContext;
import org.springframework.context.event.EventListener;
import org.springframework.kafka.annotation.KafkaListener;
import org.springframework.kafka.event.ConsumerStoppedEvent;
import org.springframework.stereotype.Component;
    
@Component
public class MyConsumer implements ApplicationContextAware {
    private ApplicationContext appCtxt;

    @Override
    public void setApplicationContext(@NotNull ApplicationContext applicationContext) throws BeansException {
        appCtxt = applicationContext;
    }

    @KafkaListener(
        topics = topicName,
        groupId = ...,
        clientIdPrefix = ...,
        containerFactory = ...
    )
    public void consume(ConsumerRecord<String, MyEvent> msg) { ... }

    @EventListener
    public void eventHandler(ConsumerStoppedEvent event) {
        log.warn("A Kafka-consumer stopped. Shutting down entire application. " + event);
        ((ConfigurableApplicationContext) appCtxt).close();
    }
}

Answer 3

I could not get Gary Russell's answer to stop my application, but found a simple solution. Maybe not entirely "boot-correct" or elegant but it does the job. What i did was inject KafkaListenerEndpointRegistry and i my SeekToCurrentErrorHandler call:

registry.stop() // Injected KafkaListenerEndpointRegistry
System.exit(SpringApplication.exit(appContext, { -1 }))