2

I am trying to create new datastreams out of one raw datastream based on certain rules available in database. Once I start the Flink job and after some, if I need to create a new datastream as per updated entries in database, how can I achieve that. So here I do not want to restart the Flink job but to create new datastream dynamically at runtime and add it to the execution DAG.

Matthias J. Sax
  • 59,682
  • 7
  • 117
  • 137
Anand Shaw
  • 229
  • 5
  • 13

1 Answers1

1

Short answer: this is not possible at runtime.

You need to stop the job and submit a new job.

Matthias J. Sax
  • 59,682
  • 7
  • 117
  • 137
  • Can you explain why this isn't possible? Is it an architectural decision to not allow changing the jobs at runtime? – Adam Lehenbauer Jul 03 '16 at 18:00
  • It is a design decision. It is pretty complex to allow for this feature. If you want to have this feature, you might want to start a discussion on the Flink mailing list. But even if the community decides for this feature, I expect it to take months until it would be available. – Matthias J. Sax Jul 03 '16 at 20:05