1

I need to implement a logic similar to session windows using processor API in order to have a full control over state store. Since processor API doesn't provide windowing abstraction, this needs to be done manually. However, I fail to find the source code for KStreams session window logic, to get some initial ideas (specifically regarding session timeouts).

I was expecting to use punctuate method, but it's a per processor timer rather than per key timer. Additionally SessionStore<K, AGG> doesn't provide an API to traverse the database for all keys.

[UPDATE]

As an example, assume processor instance is processing K1 and stream time is incremented which causes the session for K2 to timeout. K2 may or may not exist at all. How do you know that there exists a specific key (like K2 when stream time is incremented (while processing a different key)? In other words when stream time is incremented, how do you figure out which windows are expired (because you don't know those keys exists)?

Majid Azimi
  • 5,575
  • 13
  • 64
  • 113

1 Answers1

2

This is the DSL code: https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSessionWindowAggregate.java -- hope it helps.

It's unclear what your question is though -- it's mostly statements. So let me try to give some general answer.

In the DSL, sessions are close based on "stream time" progress. Only relying on the input data makes the operation deterministic. Using wall-clock time would introduce non-determinism. Hence, using a Punctuation is not necessary in the DSL implementation.

Additionally SessionStore<K, AGG> doesn't provide an API to traverse the database for all keys.

Sessions in the DSL are based on keys and thus it's sufficient to scan the store on a per-key basis over a time range (as done via findSessions(...)).

Update:

In the DSL, each time a session window is updated, as corresponding update event is sent downstream immediately. Hence, the DSL implementation does not wait for "stream time" to advance any further but publishes the current (potentially intermediate) result right away.

To obey the grace period, the record timestamp is compared to "stream time" and if the corresponding session window is already closed, the record is skipped (cf. https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/kstream/internals/KStreamSessionWindowAggregate.java#L146). I.e., closing a window is just a logical step (not an actually operation); the session will still be stored and if a window is closed no additional event needs to be sent downstream because the final result was sent downstream in the last update to the window already.

Retention time itself must not be handled by the Processor implementation because it's a built-in feature of the SessionStore: internally, the session store maintains so-called "segments" that store sessions for a certain time period. Each time a put() is done, the store checks if old segments can be dropped (based on the timestamp provided by put()). I.e., old sessions are deleted lazily and as bulk deletes (i.e., all session of the whole segment will be deleted at once) as it's more efficient than individual deletes.

Matthias J. Sax
  • 59,682
  • 7
  • 117
  • 137
  • Let me rephrase the question: Assume processor instance is processing `K1` and stream time is incremented which causes the session for `K2` to timeout. `K2` may or may not exist at all. How do you know that there exists a specific key (like `K2` when stream time is incremented (while processing a different key)? In other words when stream time is incremented, how do you figure out which windows are expired (because you don't know those keys exists)? – Majid Azimi Apr 11 '20 at 09:43
  • When `observedStreamTime` is incremented by other keys and no record comes for `K2`, then based on the implementation you provided, session window for `K2` never expires (although stream time is getting incremented) which is very unintuitive implementation. There is a single instance of stream time which essentially means when it's incremented, all expired windows has to timeout no matter we receive any further record for them or not. – Majid Azimi Apr 11 '20 at 10:02
  • 1
    Updated my answer. Let me know if this clarifies how it works? – Matthias J. Sax Apr 11 '20 at 19:18