Spring Cloud Stream: How to get KafkaEvents like ListenerContainerIdleEvent?

Question

We are using the functional programming model of Spring Cloud Stream and the Apache Kafka Streams binder. Unfortunately, we are currently not able to access the KafkaEvents - in particular the ListenerContainerIdleEvent.

@EventListener
public void events(KafkaEvent event) {
    log.info(event);
}

Our topology looks like this:

@Configuration
public class CodeObjectTopology {

    @Bean
    public Consumer<KStream<String, CodeObject>> codeObject(KafkaProtobufSerde<CodeObject> codeObjectSerde) {

        return input -> input
            .process((ProcessorSupplier<String, CodeObject, String, CodeObject>) DataProcessor::new);
    }
}

And our application.yaml looks like this:

spring:
  cloud:
    function:
      definition: codeObject
    stream:
      events:
        enabled: true
      kafka.streams:
        binder:
          application-id: app-id
          deserializationExceptionHandler: logAndFail
          configuration:
            default.key.serde: org.apache.kafka.common.serialization.Serdes$StringSerde
            default.value.serde: io.confluent.kafka.streams.serdes.protobuf.KafkaProtobufSerde
            schema.registry.url: xxx
            auto.offset.reset: earliest
            idle-event-interval: 5000
          functions:
            codeObject:
              application-id: input-topic.v1
              configuration:
                idle-event-interval: 5000
        bindings:
          codeObject-in-0:
            consumer:
              destination-is-pattern: false
              idle-event-interval: 5000
      bindings:
        codeObject-in-0:
          destination: input-topic.v1
          consumer:
            auto-startup: true
       function:
        definition: codeObject

If we try changing the KafkaEvent to SpringApplicationEvent we are getting the corresponding events.

Ultimately we want to be able to tell when a Kafka Streams function is idle (or has lag of 0).

Gary Russell · Accepted Answer · 2023-02-28T17:51:18.470

1

Spring is not involved at runtime for Kafka Streams (KStream), only the initial setup; there is no equivalent of spring events for Kafka streams (as far as I know). There is no listener container; you probably need to implement your own mechanism.

For the message channel binder, add a ListenerContainerCustomizer bean:

@Bean
ListenerContainerCustomizer<AbstractMessageListenerContainer<byte[], byte[]>> cust() {
    return (container, group, dest) -> container.getContainerProperties().setIdleEventInterval(5000L);
}

But that doesn't apply to KStream consumers.

edited Feb 28 '23 at 17:51

answered Feb 28 '23 at 17:45

Gary Russell

166,535
14
146
179

Thank you for your answer. As I am dealing with KStream consumers, is there another way to find out if the consumer lag is 0 (or something equivalent that tells us that the "initial load" has finished)? – manuelgr Feb 28 '23 at 19:09
The goal of our application is to load this topic into an Interactive Queries Store. When this has "finished" (there are very few updates to the topic) and the consumer lag is 0 or some kind of idle-event happened, we would then start all the other KStream consumers and aggregate the data using the Interactive Queries Store. But without having (a snapshot) of the complete data in the Interactive Queries store, we would produce wrong data as the results would be empty since the store is still filling up. – manuelgr Feb 28 '23 at 19:18
Another approach we thought of is to use a GlobalKTable as input binding in our aggregation KStream function and use this GlobalKTable as a state store in a Processor. But we are not sure if we get the guarantee from spring-cloud-stream that the GlobalKTable is fully built before the KStream function starts aggregating. I hope my comments made sense. – manuelgr Feb 28 '23 at 19:21
1

One easy solution is to use a task scheduler to schedule a task to run after no records are received for some time. When a new record is received, cancel the previous task and schedule a new one. Or simply keep track of when the last record was received and periodically examine it to see if some time has elapsed. – Gary Russell Feb 28 '23 at 20:40
Thank you again Gary for taking time to help me out. :) I will try to implement such a logic tomorrow. – manuelgr Feb 28 '23 at 21:25
No problem; the second option is what the listener container does. – Gary Russell Feb 28 '23 at 21:29

Spring Cloud Stream: How to get KafkaEvents like ListenerContainerIdleEvent?

1 Answers1