0

I am totally noob to Apache flink. Just trying to get my hands dirty. I have the following scenario.

  1. Datastream for Events
  2. Datastream for Events
  3. Datastream for rules
  4. Combined these two datastreams based on ruleID

Now I have a datastream of tuple3 which looks like <ruleId, Rule, Event>. These rules are SQL queries which I want to run on the Event.

I was going through concept of Dynamic Tables and Flink SQL. I am not sure how to process further. Could someone please help me out with this?

Malte Hartwig
  • 4,477
  • 2
  • 14
  • 30
Sharath Kumar Rajanna
  • 1

1 Answers1

0

Flink can execute SQL queries on data streams. A SQL is translated into a stream processing job and executed. You can maybe use Flink SQL to join the two event streams (Flink 1.4.0 supports windowed stream joins).

You cannot easily use Flink SQL to evaluate your rules, because each rule would translate to an independent streaming job. So a rule message would be processed by a job and trigger the execution of another job.

Fabian Hueske
  • 18,707
  • 2
  • 44
  • 49
  • Here the rules are the SQL queries that needs to be run on the Event (). So I am not evaluating my rules by sql. – Sharath Kumar Rajanna Feb 09 '18 at 15:05
  • I'm sorry, I don't think I understand this correctly. Can you add an example rule to your question? – Fabian Hueske Feb 09 '18 at 15:08
  • Okay Cool! So My DataStream> Looks like this: 1) Rule - "Select * from Event" 2) Event - "{"key1":"value1", "key2":"value2"}" So the rule is the Sql Query and My event is a JSON that looks like above. I need to run this query against that json using apache flink. – Sharath Kumar Rajanna Feb 09 '18 at 17:39
  • So what you want to start SQL queries based on rule events in a stream? I would have a SQL query that performs the join and writes the result to Kafka. Then you need a service that connects to the rule stream and starts a Flink SQL query that reads from the Kafka topic for every incoming rule. – Fabian Hueske Feb 09 '18 at 18:04