0

I am using Event time semantics in my Flink application (version 1.11.1) which is running in AWS - kinesis analytics. This application has source as kinesis stream and sink as Postgres. Checkpointing interval is 10 seconds as DB sink is triggered on notifyCheckpointComplete(). I am using multiple CoProcessFunction along with ValueState to connect different streams before I sink it to Postgres.

Observation is the Checkpointed Data Size is growing over the period of time whereas thread count and Heap memory utilization remains constant. CPU utilization does not go beyond 30 percent. I am hoping the checkpoint data size to plateau eventually.

enter image description here

While going through flink documentation on State TTL, it seems that currently state ttl only supported for Processing time semantics - State Time-To-Live (TTL)

What is the way forward for Event time based Flink application?

Swapnil Khante
  • 547
  • 2
  • 10

1 Answers1

0

You can still use State TTL. It's just that the state retention interval has to be expressed in terms of wall-clock time, rather than being related to the timestamps in the events.

But if you want more control over when and how the state is cleared, you can use timers in your CoProcessFunctions to trigger explicit calls to clear.

David Anderson
  • 39,434
  • 4
  • 33
  • 60
  • Thanks David for quick response. However I am using state Ttl with CoProcess function but it could not improve the checkpoint size much. After drilling down into individual tasks, the timeWindow() function on keyed stream seems to be consuming most of the size. How to deal with state for such scenarios ? – Swapnil Khante Mar 04 '21 at 04:28
  • See https://stackoverflow.com/questions/66472206/why-are-my-flink-windows-using-so-much-state/66472207#66472207. – David Anderson Mar 04 '21 at 09:31