Questions tagged [flink-checkpoint]
9 questions
1
vote
1 answer
Flink upload checkpoint to AWS S3 ERROR: Forbidden Status Code: 403
I deployed a flink application on a Kind cluster(https://kind.sigs.k8s.io/)(1 master & 2 worker nodes) using a yaml file.
As I want to upload flink checkpoint to a S3 bucket, I manually created testBucket/checkpoints, but I got this error from Job…
aqqqqqq
- 15
- 3
1
vote
1 answer
Apache Flink - streaming app doesn't start from checkpoint after stop and start
I have the following Flink streaming application running locally, written with the SQL API:
object StreamingKafkaJsonsToCsvLocalFs {
val brokers = "localhost:9092"
val topic = "test-topic"
val consumerGroupId = "test-consumer"
val…
Gabio
- 9,126
- 3
- 12
- 32
0
votes
1 answer
Incremental Checkpoint Data Size ( Flink)
we are using Flink and we have enabled the incremental checkpointing by setting state.backend.incremental=true. We are using rocksdb as state backend. With incremental checkpointing, we expect "Checkpointed Data Size" to be smaller than "Full…
vikas J
- 11
- 1
0
votes
0 answers
Old Flink checkpoints not always correctly deleted
Summary
We are using Flink 1.15.1 and have long-running stateful Flink jobs ingesting data from Kafka topics. They are configured to write checkpoints, with a RocksDB backend on S3.
We have noticed that sometimes the S3 "folders" for some…
Colin Smetz
- 131
- 7
0
votes
1 answer
Flink: Key serializer used in Java DataSet API incompatible with that used in Scala DataStream API
Our Flink code (currently, using Flink 1.12) is written in Scala and generally contains a bunch of keyed time windows and process functions as operators. We have externalized savepoint storage to allow for state restoration upon failure. Recently,…
vanrzk
- 11
- 3
0
votes
2 answers
upgrade Flink minor version and restore from checkpoint
From official doc, it says Flink support minor version upgrade - restoring a snapshot taken with an older minor version of Flink (1.x → 1.y)..
Q1. Does it means I can upgrade Flink version of my job in following way:
Stop job running with Flink…
Shenjiaqi
- 11
- 2
0
votes
1 answer
Flink Incremental CheckPointing Compaction
We have a forever running flink job which reads from kafka , creates sliding time windows with (stream intervals :1hr , 2 hr to 24 hr) and (slide intervals : 1 min , 10 min to 1 hours).
basically its :…
Pritam Agarwala
- 1
- 2
0
votes
1 answer
How to Control Size of Flink Checkpoints
I am running a simple Flink aggregation job which consumes from Kafka and applies multiple windows(1 hr, 2 hr...upto 24 hours) with specific sliding interval and does the aggregation on windows. Sometimes the job restarts and we loose the data as it…
Pritam Agarwala
- 1
- 2
0
votes
1 answer
Retain Flink Checkpoint on cancellation
I'm using Flink 1.15.0 and I want to keep triggered checkpoint when job is cancelled.
Flink indicates to set ExternalizeCheckpointCleanup mode in this way
env.getCheckpointConfig().setExternalizedCheckpointCleanup(
…
Vin
- 701
- 1
- 9
- 30