How to keep flink batch job running continuously on local

Question

I am practicing file reading through the flink batch processing mechanism on a Windows 10 machine.

I downloaded flink-1.7.2-bin-hadoop24-scala_2.12.tgz from flink's official site and executed start-cluster.bat .

I uploaded the jar though Flink's UI and was able to execute the job but the job finished in a matter of seconds.

I want to keep the job running continuously so that I can test my use case .

Can you guide my possible ways to achieve this?

@DavidAnderson: Job should be running continously untill and unless i stop it manually. — MiniSu, Jul 26 '21 at 14:39

score 1 · Answer 1 · answered Jul 26 '21 at 14:45

1

In Flink, batch jobs run until all of their input has been processed, at which point they have finished and are terminated. If you want continuous processing, then you should either

use some deployment automation (outside of Flink) to arrange for new batch jobs to be created as needed, or
implement a streaming job

In your case it sounds like you might be looking for the FileProcessingMode.PROCESS_CONTINUOUSLY option on StreamExecutionEnvironment.readfile -- see the docs for more info.

answered Jul 26 '21 at 14:45

David Anderson

39,434
4
33
60

Do you have any example program to refer it? – MiniSu Aug 12 '21 at 07:29
To the point number 2 "implement a streaming job" : What sort of implementation can be done.Can you little more elaborate? – MiniSu Aug 12 '21 at 07:44

How to keep flink batch job running continuously on local

1 Answers1