End-of-window outer join with KafkaStreams

Question

I have a Kafka topic where I expect messages with two different key types: old and new. i.e. "1-new", "1-old", "2-new", "2-old". Keys are unique, but some might be missing.

Now using Kotlin and KafkaStreams API I can log those messages with have same key id from new and old.

    val windows = JoinWindows.of(Duration.of(2, MINUTES).toMillis())

    val newStream = stream.filter({ key, _ -> isNew(key) })
            .map({key, value ->  KeyValue(key.replace(NEW_PREFIX, ""), value) })

    val oldStream = stream.filter({ key, _ -> isOld(key) })
            .map({key, value ->  KeyValue(key.replace(OLD_PREFIX, ""), value) })

    val joined = newStream.join(oldStream,
            { value1, value2 -> "$value1&$value2" }, windows)

    joined.foreach({ key, value ->
        log.info { "JOINED $key : $value" }
    })

Now I want to know new/old keys which are missing in time window for some reason. Is it possible to achieve with KafkaStreams API?

In my case when key "1-old" is received and "1-new" is not within 2 minutes only in this case I want to report id 1 as suspicious.

Matthias J. Sax · Accepted Answer · 2018-01-14T19:47:01.113

6

The DSL might not give you what you want. However, you can use Processor API. Having say this, the leftJoin can actually be used to do the "heavy lifting". Thus, after the leftJoin you can use .transform(...) with an attached state to "clean up" the data further.

For each old&null record you receive, put it into the store. If you receive a later old&new you can remove it from the store. Furthermore, you register a punctuation and on each punctuation call, you scan the store for entries that are "old enough" so you are sure no later old&new join result will be produced. For those entries, you emit old&null and remove from them from the store.

As an alternative, you can also omit the join, and do everything in a single transform() with state. For this, you would need to KStream#merge() old and new stream and call transform() on the merged stream.

Note: instead of registering a punctuation, you can also put the "scan logic" into the transform and execute it each time you process a record.

edited Jan 14 '18 at 19:47

answered Jan 12 '18 at 23:29

Matthias J. Sax

59,682
7
117
137

1

Why should I choose this approach over groupby and subsequential windowed reduce? – sumek Feb 21 '19 at 15:08
2

Multiple reasons: (1) joins use sliding windows while aggregation use hopping windows (so it's different semantics) (2) reduce() is evaluated continuously, too, thus you can hit the exact some issue and would need to use suppress() plus a downstream operator that computes the actual join. (3) in combination with point (2), you would need to put all raw data as result of `reduce` (ie, use `aggregate` and return a list of raw records). This blows up the message size and might not work for larger windows. – Matthias J. Sax Feb 21 '19 at 18:08
By `a downstream operator that computes the actual join` you've meant doing leftJoin as a separate step? We were thinking of doing the leftJoin first, only then doing the group by + reduce with a window length supression. In terms of windows I guess we could also use SessionWindow. As far I understand using SessionWindow would solve the 1) problem. Does such approach make sense? – sumek Feb 22 '19 at 12:24
1

I see -- you still have the same issue that you may create potentially big messages in a window -- I am also not sure, if you can distinguish result records correctly: assume you have `(timestamp,key,value)` pairs with join window size `5ms`: left input `(9,A,a)` -> output `(9,A,)`, left input `(10, A, a)` -> output `(10, A, )`, right input `(15,A,x)` -> output `(15,A,)`, how do you know that this replaces the second record but not the first? Also note, if right input would have ts=14, two record would be emitted on the join. Thus, I have doubt that it would work. – Matthias J. Sax Feb 23 '19 at 00:46

score 2 · Answer 2 · answered Jan 10 '18 at 23:39

2

If I understand your question correctly you only want to report id's as suspicious when there is an "old" without a corresponding "new" within the 2-minute window.

If that's the case you'll want to use a left join :

val leftJoined = oldStream.leftJoin(newStream,...).filter(condition where value expected from "new" stream is null);

HTH

answered Jan 10 '18 at 23:39

bbejeck

1,310
8
7

Your assumption regarding suspicious id is correct. However, if I leftJoin an old stream to a new stream for those matching ids I would get `old&null` and then `old&new` values in the stream for the same key. (I assume old key-value always arrives first). And if I filter values by contains("&null") I would get all ids as suspicious. – Nikolay Kuznetsov Jan 11 '18 at 11:23

score 1 · Answer 3 · answered Apr 13 '18 at 16:27

Looks like what you were looking for. Kafka Streams left outer join on timeout

Eliminates the lack of sql-like left join semantic in kafka streams framework. This implementation will generate left join event only if full join event didn't happen in join window duration interval.

End-of-window outer join with KafkaStreams

3 Answers3

Linked

Related