2

is it possible to query an existing StreamTable externally, without uploading a .jar get the execution environment and retrieve the table environment? I had waited for Apache Flink Table 1.4 release, because of its dynamic (continuous) table features. I expected something else, I thought it would be possible to alter the table at runtime and modify its parameters. In order to do some live queries, instead of defining (continuous or append-only) database views on top of a data stream. I know I could export my table into some database and query this database dynamically using SQL, but this is kind of awkward behavior. The beauty of Flink is that everything is in real-time and everything is a stream, so is it possible to query a Flink table in real time from some external program?

Fabian Hueske
  • 18,707
  • 2
  • 44
  • 49
lkaupp
  • 551
  • 1
  • 6
  • 17

1 Answers1

4

No, this is not supported at the moment.

There has been some work on storing the result table of a streaming query as queryable state. This would allow for point (key look-up) queries on a fixed key attribute. This feature might become available with Flink 1.5.

There are not concrete plans to support SQL queries on a dynamic table produced by a streaming SQL (or Table API) query. You would have to emit the table to a RDBMS and query the data from there.

Fabian Hueske
  • 18,707
  • 2
  • 44
  • 49
  • Well this leads to data duplication. It's ok in case of automatic data cube creations and transformation steps. Do you know the reason why this is so difficult to implement in Flink? The jobmanager should know the table environment and the table is only a view defined on top of a stream (which is already created by sql-like syntax). Why it isn't possible to use that sql engine and open some rest endpoint in jobmanager to send some sql queries to already created named views? (some RDBMS featured rest endpoint, so e.g. hybernate could treat this as some reading endpoint) – lkaupp Oct 25 '17 at 11:07
  • The data table resides in Flink's state backend which is distributed across the cluster and only accessible by the streaming job (except for key lookup queries with the queryable state feature). Implementing such a feature would require changes all over the place and be a major engineering effort. Right now, the community invests its time in extending the SQL coverage. – Fabian Hueske Oct 25 '17 at 14:59
  • Thank you for this statement :) maybe i'll use some timeseries db for this kind of queries – lkaupp Nov 03 '17 at 15:18