1

My requirement is to get aggregated data from kafka stream with 1 min tumbling window but as commit interval is not in sync with window size it is getting committed even before the proper aggregation.

kafka stream properties:

        props.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG,//server);
        props.put(StreamsConfig.APPLICATION_ID_CONFIG, "ID");
        props.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 60000L);
        props.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
        props.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.String().getClass());

Kafka stream:

        KTable<Windowed<String>, String> kTableMetric = inputStream.flatMap((key, value) -> {
            //aggregation logic
        }).groupByKey(Serialized.with(Serdes.String().getClass(), Serdes.String().getClass()))
                .windowedBy(TimeWindows.of(60000L)).reduce(Count::merge);

Can somebody help me on this?

Gary Russell
  • 166,535
  • 14
  • 146
  • 179

0 Answers0