1

We have an Apache Beam pipeline and need to run multiple BeamSql queries. The queries are not known at the pipeline construction time, but will be known when the pipeline is running. The queries will be updated periodically. Is this possible with BeamSql? We are using the Google Dataflow runner.

Anna Kasikova
  • 37
  • 1
  • 4
  • Thank you for your reply. Yes, the main input to the pipeline are logs and the secondary input are arbitrary SQL queries to run on the logs. Can you elaborate on what you mean by launching "sub" pipelines? – Anna Kasikova May 13 '18 at 07:22

1 Answers1

0

I'm afraid this is not possible with Beam SQL. The query is translated to a pipeline structure at construction time.

Are you receiving SQL statements as inputs to your pipeline, then? One possibility is launching "sub" pipelines from within your pipeline.

Kenn Knowles
  • 5,838
  • 18
  • 22