0

I need to catch Kafka warnings like "Bootstrap broker localhost:9092 (id: -1 rack: null) disconnected". I've found the way to achieve this using @EventListener and NonResponsiveConsumerEvent. The issue is Spring Kafka doesn't send this event at all. I use Spring Boot 2.7.0 and Spring Kafka 2.8.6. When I run application ConsumerStartedEvent is triggered, so events work in general. But when I shutdown Docker Kafka container I can see warnings, but event is not triggered.

WARN Connection to node -1 (localhost/127.0.0.1:9092) could not be established. Broker may not be available.

My configuration:

@Autowired
@Bean
public ConcurrentKafkaListenerContainerFactory<String, String> kafkaListenerContainerFactory(
        KafkaProperties properties
) {
    ConcurrentKafkaListenerContainerFactory<String, String> factory =
            new ConcurrentKafkaListenerContainerFactory<>();
    factory.setConsumerFactory(consumerFactory(properties));
    factory.getContainerProperties().setMonitorInterval(10);
    return factory;
}

and listener

    @KafkaListener(topics = "topic", groupId = "foo")
public void listenMessages(ConsumerRecord<String, TickService.TickServiceEvent> record) {
    ...
}

//works
@EventListener(ConsumerStartedEvent.class)
public void eventHandler(ConsumerStartedEvent event) {
    isConnected = true;
}

//works only for ConsumerStartedEvent
@EventListener
public void eventHandler(KafkaEvent event) {

    if(event instanceof NonResponsiveConsumerEvent) {
        isConnected = false;
    }
}

//doesn't work at all
@EventListener(NonResponsiveConsumerEvent.class)
public void eventHandler(NonResponsiveConsumerEvent event) {
    isConnected = false;
}

What have I missed? Thx!

Ievgen Rebrakov
  • 1
  • 3

1 Answers1

0

The event you expect is not correct in this context:

NonResponsiveConsumerEvent: published when the consumer appears to be blocked in the poll method.

This event is not about a connection. It is emitted when processing of records in your listener method is longer then a poll timeout allows.

The message you show is just warning in logs and unfortunately there is no way to catch such a situation from the outside. That's how Kafka Consumer works with its internal Cluster logic: it does not throw an exception in case of lost connection and just retries indefinitely: Kafka consumer reconnection after getting disconnected

You probably better look into a:

ListenerContainerIdleEvent: published when no messages have been received in idleInterval (if configured).

So, this way you can determine that your listener has not received the data for a while.

With my current debugging experience I even don't see that retries option helps somehow: the org.apache.kafka.clients.NetworkClient still tries to reconnect forever...

Artem Bilan
  • 113,505
  • 11
  • 91
  • 118
  • Thx! But documentations says: In addition, if the broker is unreachable (at the time of writing), the consumer poll() method does not exit, so no messages are received, and idle events can’t be generated. To solve this issue, the container will publish a NonResponsiveConsumerEvent if a poll does not return within 3x the pollInterval property. https://docs.spring.io/spring-kafka/docs/2.2.0.M2/reference/html/_reference.html#idle-containers – Ievgen Rebrakov Jun 06 '22 at 07:13
  • Ok. I’ll look at that closer, but probably something has changed since that old not supported already version… – Artem Bilan Jun 06 '22 at 11:57
  • `>(at the time of writing)` - that is pertinent - they fixed the client so that it now returns; the framework will (at the time of **this writing**) never issue a `NonResponsiveConsumerEvent` (unless they introduce some bug in the future that causes the client to hang). – Gary Russell Jun 06 '22 at 13:22
  • Unfortunately, the consumer client provides no API to tell us whether the broker is available or not. You can use an `AdminClient` (e.g. `describeCluster()`) to check. – Gary Russell Jun 06 '22 at 13:24