Parse SQL and Evaluate an Expression

Question

I have a SQL query that I would like to parse evaluate. I have parsed the SQL using JSQL Parser. Now, I need to evaluate the where clause in the SQL. I would like to do it in Flink as part of the filter function. Basically, stream.filter(Predicate<Row>). The Predicate<Row> is what I need to get from the evaluation of the SQL's where clause and apply it on the streaming record.

Ex: SELECT COLUMN FROM TABLE WHERE (ac IS NOT NULL AND ac = 'On')

I would like to parse the above query and given a streaming record with say ac = on, I would like to run the above expression evaluation on that record.

Any thoughts on how I can do it?

I would like to try using expression evaluation with DFS but kinda confused how to run by it. Any help is appreciated!

Answer 1

If the SQL query is known at compile time, it's more straightforward to do this by integrating Flink SQL (via the Table API) into your DataStream application. See the docs for more info and examples.

The overall approach would be to convert your DataStream into a dynamic Table (which can be done automatically if the stream is a convenient type, such as a POJO), apply the SQL query to it, and then (if necessary) convert the resulting Table back to a DataStream.

Or maybe just implement the entire application with the Table API if you don't need any of the features that are unique to DataStreams.

On the other hand, if the query is dynamic and isn't provided until runtime, you'll need to pursue something like what you've proposed. FWIW, others with similar requirements have used dynamic languages with JVM-based runtimes, such as Javascript via Rhino, or Groovy. The overall approach is to use a BroadcastProcessFunction, with the dynamic code being broadcast into the operator.