Is it possible to implement left outer join in Flink's CoGroupedStreams.coGroup()

Question

In reference to post - Flink co group outer join fails with High Backpressure

would "CalculateCoGroupFunction.coGroup" method implementation support left outer join on multiple streams (in my use case I have three stream sources that would need to be joined). I would appreciate if you could share any examples you may have on left outer joins using coGroup().

Thanks in advance!

DataStream<Message> pStream =
    stream1
    .coGroup(stream2)
    .where(obj -> obj.getid())
    .equalTo(ev -> ev.getid())
        .window(TumblingEventTimeWindows.of(Time.minutes(Constants.VALIDTY_WINDOW_MIN)))
        .evictor(TimeEvictor.of(Time.minutes(Constants.VALIDTY_WINDOW_MIN)))
    .apply(new CalculateCoGroupFunction());

Looking for a working example of coGroup implementation for left outer joins on multiple streams ( > 3 streams)

So you have one stream that is the "right side", and > 1 stream for the left side? — kkrugler, Feb 14 '23 at 17:46
Hi @kkrugler, thanks for the question. left side = 1 stream, and on the right side > 1 stream. to give an analogy in sql terms that would look like ( where a1 is stream on the left side, and b1 and c1 are two streams on the right). Essentially keep all records from a1 stream SELECT * FROM a1 LEFT JOIN b1 ON foo... LEFT JOIN c1 ON bar... — Pradeep, Feb 17 '23 at 15:24

score 0 · Answer 1 · answered Feb 19 '23 at 01:56

0

You can do a union on stream 2 and 3 (right side), then coGroup on stream1. Not sure if this is what you are looking for.

stream1
    .coGroup(stream2.union(stream3))
    ...

answered Feb 19 '23 at 01:56

Sean L

48
4

score 0 · Answer 2 · answered Feb 28 '23 at 04:01

If the right side streams are the same type (contain records of the same class) then you can do a union as per @Sean L's answer above.

If not, then the standard approach is to create a class (something like Flink's Either) that can contain one of N different records, each with a different type. You first run each right side stream through a map function that creates this Either record, then you can union them all together before doing the coGroup(). You'll typically have a .getKey() method in this Either class that knows how to extract the appropriate field from the record that is in the wrapper class.

Is it possible to implement left outer join in Flink's CoGroupedStreams.coGroup()

2 Answers2