0

We have a scenario where we have to persist/save some value into the checkpoint and retrieve it back during failure recovery/application restart.

We followed a few things like ValueState, ValueStateDescriptor still not working. https://github.com/realtime-storage-engine/flink-spillable-statebackend/blob/master/flink-spillable-benchmark/src/main/java/org/apache/flink/spillable/benchmark/WordCount.java

https://towardsdatascience.com/heres-how-flink-stores-your-state-7b37fbb60e1a https://github.com/king/flink-state-cache/blob/master/examples/src/main/java/com/king/flink/state/Example.java

We can't externalize it to a DB as it may cause some performance issues. Any lead to this will be helpful using checkpoint. How to put and get back from a Checkpoint?

ouflak
  • 2,458
  • 10
  • 44
  • 49
Ajit Raju
  • 21
  • 5
  • Your application looks okay. How are you testing the checkpointing? See https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/try-flink/flink-operations-playground/#observing-failure--recovery for a tutorial explaining this. – David Anderson Jan 19 '22 at 08:18
  • is there a way to put and get some data into Checkpoint? its fine if we code as well. – Ajit Raju Jan 19 '22 at 09:00

1 Answers1

0

All of your managed application state is automatically written into Flink checkpoints (and savepoints). This includes

  • keyed state (ValueState, ListState, MapState, etc)
  • operator state (ListState, BroadcastState, etc)
  • timers

This state is automatically restored during recovery, and can optionally be restored during manual restarts.

The Flink Operations Playground shows how to work with checkpoints and savepoints, and lets you observe their behavior during failure/recovery and restarts/rescaling.

If you want to read from a checkpoint yourself, that's what the State Processor API is for. Here's an example.

David Anderson
  • 39,434
  • 4
  • 33
  • 60
  • Hi Davi, thanks for the response we have tried with Keyed State still our value is not getting stored/ retrived from checkpoint. e.g. – Ajit Raju Jan 19 '22 at 10:47
  • private transient ValueState wordCounter; wordCounter.update(currentValue); Open method wordCounter = getRuntimeContext().getState(new ValueStateDescriptor<>( "wc", TypeInformation.of(new TypeHint(){}));); – Ajit Raju Jan 19 '22 at 10:53
  • we also tried with private ListState state; and update the state in a RichFlatMapFunction still not able to get the value – Ajit Raju Jan 19 '22 at 15:09
  • What are you doing when you try to "get the value"? Are you saying that your checkpoints aren't being restored properly, or that `wordCounter.value()` isn't behaving as you expect during normal operation? – David Anderson Jan 19 '22 at 16:23
  • Keep in mind that with keyed state, the state being accessed and updated is automatically scoped to the key of the current event. – David Anderson Jan 19 '22 at 16:25
  • Hi David, yes when we try to wordCounter.value() during restart of the appliaction its getting null. checked with using both Default and RocksDb checkpointing. any suggestion. – Ajit Raju Jan 20 '22 at 17:18
  • In order for `wordCounter.value` to be non-null you must explicitly restart the application from a retained checkpoint or a savepoint, and you must be calling flatmap with a word that already been counted during the previous run. If you believe you are doing this correctly, please share a reproducible example, clearly explaining all of the steps required to experience the failure. – David Anderson Jan 20 '22 at 19:41
  • Hi David, your observation is right sending you code in two comment public class Counter extends RichFlatMapFunction { private ValueState wordCounter; public Counter() throws IOException { } @Override public void flatMap(Event value, Collector out) throws Exception { String cV = wordCounter.value(); log.info("1 " + cV); cV = cV == null ? value.getId() : "tempvalue"; wordCounter.update(cV); log.info("2" + wordCounter.value()); out.collect(cV); } – Ajit Raju Jan 21 '22 at 06:04
  • @Override public void open(Configuration config) { wordCounter = getRuntimeContext().getState(new ValueStateDescriptor<>("wc",TypeInformation.of(new TypeHint(){}))); } } when we restart the JM the value state is not getting restored. it beings with null again. – Ajit Raju Jan 21 '22 at 06:05
  • Sure, but having the right code isn't enough. You have to do the restart properly, and provide the right data. – David Anderson Jan 21 '22 at 09:38
  • Can you help us how to ensure proper restart and confg – Ajit Raju Jan 21 '22 at 11:12
  • Please look at the operations playground that I shared earlier. It shows how you can test this. – David Anderson Jan 21 '22 at 12:34