Apache Flink join different DataStreams on specific key

Question

I have two DataStreams, the first one called DataStream<String> source which receive records from a message broker, and the second one is a SingleOutputOperator<Event> events, which is the result of mapping the source into Event.class.

I have a uses cases that needs to use SingleOutputOperator<Event> events and other that uses DataStream<String> source. In one of the use cases that use DataStream<String> source, I need to join the SingleOutputOperator<String> result after apply some filters and to avoid to map the source again into Event.class as I already have that operation done and that Stream, I need to search each record into the SingleOutputOperator<String> result into the SingleOutputOperator<Event> events and the apply another map to export a SingleOutputOperator<EventOutDto> out.

This is the idea as example:

DataStream<String> source = env.readFrom(source);
SingleOutputOperator<Event> events = source.map(s -> mapper.readValue(s, Event.class));


public void filterAndJoin(DataStream<String> source, SingleOutputOperator<Event> events){
  
  SingleOutputOperator<String> filtered = source.filter(s -> new FilterFunction());
  
  SingleOutputOperator<EventOutDto> result = (this will be the result of search each record 
      based on id in the filtered stream into the events stream where the id must match and return the event if found)
.map(event -> new EventOutDto(event)).addSink(new RichSinkFunction());
}

I have this code:

filtered.join(events)
                    .where(k -> {
                        JsonNode tree = mapper.readTree(k);
                        String id = "";
                        if (tree.get("Id") != null) {
                            id = tree.get("Id").asText();
                        }
                        return id;
                    })
                    .equalTo(e -> {
                        return e.Id;
                    })
                    .window(TumblingEventTimeWindows.of(Time.seconds(1)))
                    .apply(new JoinFunction<String, Event, BehSingleEventTriggerDTO>() {
                        @Override
                        public EventOutDto join(String s, Event event) throws Exception {
                            return new EventOutDto(event);
                        }
                    })
                    .addSink(new SinkFunction());

In the above code all works fine, the ids are the same, so basically the where(id).equalTo(id) should work, but the process never reaches the apply function.

Observation: Watermark are assigned with the same timestamp

Questions:

Any idea why?
Am I explained myself fine?

score 0 · Answer 1 · answered Apr 23 '21 at 14:27

I solved the join by doing this:

SingleOutputStreamOperator<ObjectDTO> triggers = candidates
                    .keyBy(new KeySelector())
                    .intervalJoin(keyedStream.keyBy(e -> e.Id))
                    .between(Time.milliseconds(-2), Time.milliseconds(1))
                    .process(new new ProcessFunctionOne())
                    .keyBy(k -> k.otherId)
                    .process(new ProcessFunctionTwo());

Apache Flink join different DataStreams on specific key

1 Answers1