0

I want to use Kafka Processor API to process messages from Kafka. I would like to call some periodically function - something like: context.schedule(IntervalMS,punctuationType, somePunctuator), where somePunctuator perform some periodical job, but instead using interval time as trigger I would like to invoke that task after processing some number of messages

Is it possible do such triggering in Kafka streams?

Jacek Laskowski
  • 72,696
  • 27
  • 242
  • 420
Bartosz Wardziński
  • 6,185
  • 1
  • 19
  • 30

1 Answers1

1

yes, it's possible with using Kafka Streams State Store. logic depends on what exactly you need to do on reaching the number of processed messages.

if you need to propagate data to the next processor or sink node, you need to store aggregated values as a list of objects inside key-value state store. inside Processor.process(..) you put data into key-value store, and after that check whether number of items reached limit, and do required logic (like processorContext.forward(..)). please take a look at similar example here.

if you need to do some logic after reaching number and don't need values, you could store only counter, and inside Processor.process(..) increment this value.

Vasyl Sarzhynskyi
  • 3,689
  • 2
  • 22
  • 55