Fail Fast When Kafka Consumer Can't Connect

Question

For my own purposes, I need the Spring Boot application to stop if the Kafka Consumer can't connect to the broker. I mean, when Kafka Consumer trying to pool messages, we can see the following logs:

[Consumer clientId=consumer-ddddd-1, groupId=ddddd] Bootstrap broker localhost:9094 (id: -1 rack: null) disconnected
[Consumer clientId=consumer-ddddd-1, groupId=ddddd] Connection to node -1 (localhost/127.0.0.1:9094) could not be established. Broker may not be available.

It`s standard behaviour when topic or broker is not available. As a result - application will not going to stop. But I need.

I'm trying to add the following properties, but it's not work:

spring.kafka.consumer.fetch-max-wait=1000
spring.kafka.admin.fail-fast=true
spring.kafka.session.timeout.ms=1000

In generally I want to get behaviour like: IF CONSUMER CAN'T CONNECT - SHUTDOWN APPLICATION

Spring Boot version: 2.3.8.RELEASE
Kafka: spring-kafka-starter

Example of Kafka Polling:

consumer.poll(Duration.ofMinutes(5));

You want poll to fail after 1 second in case of no response? If that is the case it will never happen with this call `consumer.poll(Duration.ofMinutes(5));` since you are setting here timeout of 5 minutes. You could simply run it with `consumer.poll(1000);` and see if that works. `poll` method requires a timeout to be set while calling, so there is no additional config needed for timeout. — alxbxbx, Feb 11 '21 at 13:48
I studied approaches to how to do it, you are right. The best way is to go beyond the Duration. I will now write an answer to this question, thank you for your participation! — Oleksandr, Feb 11 '21 at 14:07

score 0 · Accepted Answer · answered Feb 11 '21 at 14:11

As one of the approaches - we can use from the comment above. Or I've just created validation based on the following code and it works

    public void validate() {
        try {
            consumer.listTopics(Duration.ofSeconds(10));
        } catch (TimeoutException e) {
            logger.error("Topics doesn't exist OR unavailable broker");
            System.exit(1);
        }
    }

score 0 · Answer 2 · answered Mar 24 '23 at 16:07

My solution to the problem:

    @Bean
    @Profile("local")
    public Boolean localKafkaAvailabilityValidator() {
        KafkaAdmin kafkaAdmin = new KafkaAdmin(kafkaProperties.buildAdminProperties());
        kafkaAdmin.setFatalIfBrokerNotAvailable(true);

        var future = CompletableFuture.runAsync(kafkaAdmin::clusterId);
        try {
            future.get(2, TimeUnit.SECONDS);
        } catch (Exception e) {
            throw new RuntimeException("Kafka is not available. Please start it before starting the application.", e);
        }

        return true;
    }

Fail Fast When Kafka Consumer Can't Connect

2 Answers2