0

I am currently exploring on enabling async checkpointing in Spark Structured streaming , but not able to find any way for the same. DataBricks is offering the same for its flavour of Spark.

Spark Structured Streaming 3.3.1 and RocksDB 7.7.3

Any suggestions on the same.

Aviral Kumar
  • 814
  • 1
  • 15
  • 40

1 Answers1

0

Shared your question with the Speedb hive on discord and here is what we have for you; From Hilik, our co-founder and chief scientist:

"Rocksdb currently does not have a mechanism for async checkpoints. The checkpoint is done by halting all the i/o flush the memtables and then use a hard link on the file system. Since this is a very destructive operation it exists on our to-do list. if you are interested please suggest a feature to the community and we will prioritize it according to the interest"

Hope this helps, let us know if you have any other questions. Join the discussions on Discord and click here for the thread about your question

Dan Carfas
  • 46
  • 2