Flink StatefulFunction reacting to a checkpoint?

Question

My stateful function is writing data to a database, but it is nondeterministic so restoring from a checkpoint can result in the database containing inconsistent data. My idea is to 'buffer' data in Flink's persistent state and only write the data to the database once a checkpoint has finished. I guess I could sort of achieve this by using context.sendAfter(Duration duration, Address address, Object input);, by setting the duration higher than the checkpointing interval.

Is there a better solution available that would enable the function to react to checkpoints explicitly?

score 1 · Accepted Answer · answered Aug 25 '20 at 13:50

1

It's planned that Stateful Functions 2.2 will support Flink datastreams as ingresses and egresses -- see https://github.com/apache/flink-statefun/pull/133 -- which should then allow you use a Flink sink connector that meets your needs. If Flink doesn't already include a suitable sink, you could implement one based on the generic two-phase commit sink (which participates in the checkpointing process).

Another option would be to somehow make it possible for stateful functions to be aware of checkpointing, but so far that hasn't been done (or even discussed, as far as I know).

answered Aug 25 '20 at 13:50

David Anderson

39,434
4
33
60

So would that make it possible write a datastream function that implements CheckpointedFunction and use it as an egress? – weequ Aug 26 '20 at 05:51
Yes, I believe so. – David Anderson Aug 26 '20 at 12:05

score 1 · Answer 2 · answered Sep 01 '20 at 12:35

The best way, at the moment, to deal with side-effects leaving a stateful function (i.e. a write to a database) is to use an exactly once egress (like Kafka). Instead of writing to the database directly from the function, emit a message to a "side effecting egress".

To complete this, you would need to write a simple program that consumes from that Kafka topic (only committed records) and write them to the database.

To simplify this, future versions of StateFun would contain a native exactly once JdbcSink.

Flink StatefulFunction reacting to a checkpoint?

2 Answers2