Flink Stateful Function Redeploy

Question

I cannot find anything around versioning issues when redeploying a Flink Stateful Function.

When I update and deploy code for a stateful function that has currently executing instances, what effect does that have on:

1. Instances currently executing, not awaiting callbacks?
2. Instances that are currently awaiting async callback (what code runs when the callback returns)?
3. If either or 1 or 2 might be executing the new code, how is the possibly different persisted state of the 2 function versions between old version and the new executing version handled?
4. If 2 does not upgrade automatically, how do I cause the new function version to run when I want this to happen for a long running function (say it was awaiting self callback on a 30 day timer and I wanted the new version to be executing on callback return)? And if I can upgrade the version, what is the method of dealing with possible additions or removal of persisted state between versions?

Answer 1

You can implement a stateful function either as an embedded function, or as a remote function.

• An embedded function is deployed and loaded directly into the Flink cluster that executed the StateFun Job. This means that in order to deploy a new version of your embedded function, you would need to stop the existing cluster, and redeploy it with the new function. In that mode:

  1. Is not relevant, since all the existing instance would be brought down.
  2. When the new version of your application would be loaded, any previously registered timers would be delivered (assuming the time had passed) and any uncompleted asynchronous operations would be delivered with the status “unknown”.

• For remote functions, there could be a brief moment in time where you would have multiple versions alive at the same time (for example: a rolling upgrade of a k8s deployment of your function) In that case, a timer might fire either on the old version or the new version (there are no async operations support for remote functions) during that window of a rolling upgrade. In any case, any state modifications would be visible in a consistent manner between the versions.

Please note that the state values themselves are Protocol Buffers messages, so as long as you would respect Protocol Buffers schema evolution rules, you would be able to read the values across versions.