0

I have a materialized view that is updated from many streams. Every one enrich it partially. Order doesn't matter. Updates comes in not specified time. Is following algorithm is a good approach:

  1. Update comes and I check what is stored in materialized view via get(), that this is an initial one so enrich and save.
  2. Second comes and get() shows that partial update exist - add next information
  3. ... and I continue with same style

If there is a query/join, object that is stored has a method that shows that the update is not complete isValid() that could be used in KafkaStreams#filter().

Could you share please is this a good plan? Is there any pattern in Kafka streams world that handle this case?

Please advice.

zashto
  • 5
  • 3

1 Answers1

0

Your plan looks good , you have the general idea, but you'll have to use the lower Kafka Stream API : Processor API.

There is a .transform operator that allow you to access a KeyValueStatestore, inside this operation implementation you are free to decide if you current aggregated value is valid or not.

Therefore send it downstream or returning null waiting for more information.

Sid
  • 331
  • 2
  • 10