0

Hi i am trying to do checkpointing in one of my flink module in which i am using CoFlatMapFunction to combine to streams if i comment out the CoFlatMapFunction checkpointing is working if uncomment again its not working. i updated the Checkpointing as this documentation in flink website in which it says for iterative streams there is an extra attribute added to force the checkpoint event after doing that also its not working please find below for the checkpoint settings

StateBackend stateBackend = new RocksDBStateBackend(path, true);

//env.enableCheckpointing(interval, CheckpointingMode.EXACTLY_ONCE);

env.enableCheckpointing(interval,CheckpointingMode.EXACTLY_ONCE,true);

 env.getCheckpointConfig().setMinPauseBetweenCheckpoints(1000);

 env.getCheckpointConfig().setCheckpointTimeout(120000);
 
 env.getCheckpointConfig().setMaxConcurrentCheckpoints(2);

 env.getCheckpointConfig().enableExternalizedCheckpoints(CheckpointConfig.ExternalizedCheckpointCleanup.RETAIN_ON_CANCELLATION);
 
 env.getCheckpointConfig().setPreferCheckpointForRecovery(true);

 env.setStateBackend(stateBackend);

I can see on of the task status finished but i am unable to see the logs since

enter image description here

YRK
  • 153
  • 1
  • 1
  • 22
  • What do you mean by "not working"? How are you making that determination? And can you share a reproducible example? – David Anderson Sep 22 '20 at 17:56
  • Please include details - what version of Flink, what exactly do you mean by "is not working", and what your workflow topology looks like (a CoFlatMapFunction doesn't create an iteration, so you shouldn't have to force checkpoint). – kkrugler Sep 22 '20 at 17:56
  • @DavidAnderson thanks for the reply, what i mean "not working" i don't see checkpoint files not creating in the checkpointing location, sorry that i can't provide the code base but what i observed in flink dashboard is one of the task "source collection task" finished i am attaching the dashboard image to question – YRK Sep 23 '20 at 12:22

1 Answers1

2

I believe the reason for this is FLINK-2491: checkpointing only works if all operators/tasks are still running.

You should replace the source that is injecting some data from a Collection with some other source that won't just instantly transition to being finished, perhaps a custom source that keeps the source alive once it runs out of data to emit, but doing nothing.

David Anderson
  • 39,434
  • 4
  • 33
  • 60