0

Say I have a flink job processing a data flow like 1, 2, control_flag, 3... When control_flag is met, the job should be stopped with savepoint and the following messages 3... should neither be processed or dropped. When centern actions are taken outside the flink and the job is restarted from savepoint, the job should go on process the following messages. However, if the job hangs with a sleeping loop inside the process operator to prevent the following messages to be processed, it can not be stopped with savepoint using flink api. So how do I stop the job at the position of control_flag and let the job to be restarted with the position next to it?

David
  • 67
  • 1
  • 6

1 Answers1

0

Some suggestions can be found here.

There are a few possible ways that it can be done, but I think since You want to keep state between the runs, the best idea would be to have an operator that :

  • If the flag stop_execution is false, processes data and outputs that for the downstream operators.

  • If the flag stop_execution is true, it adds the data it receives to list state.

  • If it receives the control_flag it emits side output meaning that job should be stopped.

Now it's up to You to listen to the side output, this can be either external service that reads data from Kafka and executes correct REST calls to stop given job or anything else You want.

Dominik Wosiński
  • 3,769
  • 1
  • 8
  • 22
  • It's possible to store the records coming after stop_execution into list state and consume them later. But if there are many incoming records, it may take too much memory. Another idea is to keep the operator in sleeping loop, while the external service finishes its job; then the operator exits the loop and resumes the record processing. The problem of this idea is similar: when there are many records, the checkpoint may fails too many times that the job is failed, so the job parameters need to be carefully set. – David Dec 14 '22 at 03:23
  • If You worry about too state taking too much memory You can use RocksDB, which prevents this issue entirely. Another thing You may consider is to modify `Source` to stop processing input data when it sees the flag, but that's trickier and requires more careful thinking. – Dominik Wosiński Dec 15 '22 at 00:26