1

I need to re-sequence the data that comes from two topics (merged using an outer join). Is it good practice to use a StateStore to keep the latest sequence and modify the downstream stream value with the re-sequenced message.

Simplified problem :

(seq from topic A, seq from topic B) -> new seq to output (keeping the current sequence in the StateStore)

(10,100) -> 1

(11,101) -> 2

(12,102) -> 3

(...,...) -> ...

The new sequence would be stored as value for the key "currentSeq" in the stateStore. The sequence will be incremented on each message and stored back to the stateStore.

Tony
  • 1,214
  • 14
  • 18
  • I cannot completely follow. However, using a `StateStore` within `process()` or `transform()` is absolutely possible. And you can store anything there you like. – Matthias J. Sax Nov 28 '16 at 17:47
  • @MatthiasJ.Sax The important is that I need to "spit" out a new sequence number in the downstream process. And I want to get the sequence number from the StateStore, increment it and put it back to the store for next message. On some conditions that sequence number might reset to one. – Tony Nov 28 '16 at 18:02

1 Answers1

2

You should use Processor API with a registered (maybe custom) state.

You can also mix-and-match the Processor API with DSL using process(), transform() or transformValue() and reference a state store (by name).

See

Jacek Laskowski
  • 72,696
  • 27
  • 242
  • 420
Matthias J. Sax
  • 59,682
  • 7
  • 117
  • 137