0

I have 3 jobs that read from the same input stream. Each gives a different output How do I combine the results from different Jobs and create a single JSON string

Example: {"key":"input_msg", "result_1":"job1_result",...}

I am hoping to avoid querying a DB, as if I scale my jobs to a huge number that will have a negative impact.

Sub Zero
  • 69
  • 11

2 Answers2

1

Yes that is possible

available_topics = List("topic_1", "topic_2")


var streams = collection.mutable.Map[String,  DataStream[String]]()
for(a <- 0 until available_topics.size){
  streams += (available_topics(a) -> env.addSource(new FlinkKafkaConsumer09(available_topics(a), new SimpleStringSchema(), properties)).map(x => someFunctionThatS(x)))
}
Ideas frontier
  • 70
  • 9
0

You could combine all three jobs into one and then join the results of the three parts to form the joined JSON result.

Till Rohrmann
  • 13,148
  • 1
  • 25
  • 51