I am trying to use the job listener to execute dedicated actions at the beginning and the completion of my flink job.
However, the onJobSubmitted and onJobExecuted are being executed at the beginning of my job.
Below are the driver logs:
2023-06-14 06:44:00,377 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Starting execution of job 'Flink Streaming Job' (097ed72406b48a3952ad60d463bbf63b) under job master id 00000000000000000000000000000000.
2023-06-14 06:44:00,378 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Connecting to ResourceManager akka.tcp://flink@vdavtrans010dsy:6123/user/rpc/resourcemanager_*(00000000000000000000000000000000)
2023-06-14 06:44:00,378 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - Resolved ResourceManager address, beginning registration
2023-06-14 06:44:00,379 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Registering job manager 00000000000000000000000000000000@akka.tcp://flink@vdavtrans010dsy:6123/user/rpc/jobmanager_4 for job 097ed72406b48a3952ad60d463bbf63b.
2023-06-14 06:44:00,379 INFO org.apache.flink.runtime.resourcemanager.StandaloneResourceManager [] - Registered job manager 00000000000000000000000000000000@akka.tcp://flink@vdavtrans010dsy:6123/user/rpc/jobmanager_4 for job 097ed72406b48a3952ad60d463bbf63b.
2023-06-14 06:44:00,380 INFO org.apache.flink.runtime.jobmaster.JobMaster [] - JobManager successfully registered at ResourceManager, leader id: 00000000000000000000000000000000.
2023-06-14 06:44:00,380 INFO org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager [] - Received resource requirements from job 097ed72406b48a3952ad60d463bbf63b: [ResourceRequirement{resourceProfile=ResourceProfile{UNKNOWN}, numberOfRequiredSlots=2}]
2023-06-14 06:44:00,430 INFO MyMain [] - Executing onJobSubmitted
2023-06-14 06:44:00,447 WARN org.apache.flink.runtime.resourcemanager.slotmanager.DeclarativeSlotManager [] - Could not fulfill resource requirements of job 097ed72406b48a3952ad60d463bbf63b. Free slots: 0
2023-06-14 06:44:00,617 INFO MyMain [] - Executing onJobExecuted
2023-06-14 06:44:10,511 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Created execution graph 067440632cd6a3ca9a42b5402d3fd825 for job 097ed72406b48a3952ad60d463bbf63b.
2023-06-14 06:44:10,511 INFO org.apache.flink.runtime.scheduler.adaptive.AdaptiveScheduler [] - Running initialization on master for job Flink Streaming Job (097ed72406b48a3952ad60d463bbf63b).
2023-06-14 06:44:10,512 INFO org.apache.flink.runtime.scheduler.adaptive.AdaptiveScheduler [] - Successfully ran initialization on master in 0 ms.
2023-06-14 06:44:10,523 INFO org.apache.flink.runtime.scheduler.adapter.DefaultExecutionTopology [] - Built 1 new pipelined regions in 0 ms, total 1 pipelined regions currently.
2023-06-14 06:44:10,523 INFO org.apache.flink.runtime.scheduler.adaptive.AdaptiveScheduler [] - No state backend has been configured, using default (HashMap) org.apache.flink.runtime.state.hashmap.HashMapStateBackend@3f9c7abe
2023-06-14 06:44:10,523 INFO org.apache.flink.runtime.state.StateBackendLoader [] - State backend loader loads the state backend as HashMapStateBackend
2023-06-14 06:44:10,523 INFO org.apache.flink.runtime.scheduler.adaptive.AdaptiveScheduler [] - Using job/cluster config to configure application-defined checkpoint storage: org.apache.flink.runtime.state.storage.FileSystemCheckpointStorage@dcc36709
2023-06-14 06:44:10,524 INFO org.apache.flink.runtime.checkpoint.CheckpointCoordinator [] - No checkpoint found during restore.
2023-06-14 06:44:10,524 INFO org.apache.flink.runtime.executiongraph.ExecutionGraph [] - Source: Custom source -> Sink: Writer (1/1) (067440632cd6a3ca9a42b5402d3fd825_cbc357ccb763df2852fee8c4fc7d55f2_0_0) switched from CREATED to SCHEDULED.
Do you have any idea why I am facing this behavior ?
I have analyzed the logs to understand what's happening. I am pretty sure this is related to adaptive scheduler which stop and restart the job from last checkpoint
