1

I am new to flink and want to understand how to run my use case with FLINK: Application has three input data source a) historical data b) get all the live events from kafka c) get the control event that will have a trigger condition

since the application is dealing with historical data so I thought that I will merge historical data and live data and will create a table on that stream.

To trigger the event we have to write the SQL query with help of control event that is the input source and that holds the where clause.

My problem is to build the SQL query as data is in Stream and when I do something like

DataStream<ControlEvent> controlEvent
controlEvent.map(new FlatMapFunction(String, String)
{
   @override
   public String flatMap(String s, Collector<String> coll)
   {
     tableEnv.execute("select * from tableName");   /// throw serialization exception
   }
});

it throws not serialization exception Localexecutionenvironment

Hamada
  • 1,836
  • 3
  • 13
  • 27
Ashutosh
  • 33
  • 8

1 Answers1

0

That sort of dynamic query injection is not (yet) supported by Flink SQL.

Update:

Given what you've said about your requirements -- that the variations in the queries will be limited -- what you might do instead is to implement this using the DataStream API, rather than SQL. This would probably be a KeyedBroadcastProcessFunction that would hold some keyed state and you could broadcast in the updates to the query/queries.

Take a look at the Fraud Detection Demo as an example of how to build this sort of thing with Flink.

David Anderson
  • 39,434
  • 4
  • 33
  • 60
  • Thanks David for your quick reply. would you like to suggest any other approach to achieve this goal? – Ashutosh Jul 02 '20 at 04:46
  • What are the queries like, and how much variation is required? – David Anderson Jul 02 '20 at 07:33
  • not expecting much variation, query will be something like SELECT * FROM EventMessages where referencData = ? and businessDate < or > ? .. and some time there will be self join required. 1) simple sql query with 2-3 where clause field 2) self join type sql query – Ashutosh Jul 02 '20 at 08:50
  • And volume of these controlSet will be approx 2-5K. – Ashutosh Jul 02 '20 at 08:57
  • David, So you are suggesting to come out from sql approach as BroadCastProcessFunction will also not allow to execute the sql query due to same reason (Serialization issue ).. or I misunderstood your statement? – Ashutosh Jul 03 '20 at 13:39
  • That's right: doing this with SQL is not possible, so I suggested an alternative. With SQL the query has to be fixed at compile time. – David Anderson Jul 03 '20 at 13:47