I am using flink-1.4.2 with scala and RocksDB is used for state backend but I am not getting sst files in taskmanager.
Asked
Active
Viewed 505 times
1 Answers
2
Have you set state.backend.rocksdb.localdir? If not, the SST files will be stored wherever io.tmp.dirs points to, which defaults to /tmp in Linux systems.
David Anderson
- 39,434
- 4
- 33
- 60
-
Hi David, I have not set state.backend.rocksdb.localdir and so, I have checked /tmp but I do not get any sst files – utkarsh baranwal Oct 18 '19 at 11:48
-
I would then want to see how you have configured RocksDB to be the state backend, and also verify that your job has state that can be stored in RocksDB. Keep in mind that non-keyed state is never stored in RocksDB, and by default timers are also on-heap. – David Anderson Oct 18 '19 at 12:05
-
env.setStateBackend(new RocksDBStateBackend("file:///mnt/flink_rocks/checkpoints", false)) – utkarsh baranwal Oct 18 '19 at 12:29
-
I have used this code to set Rocksdb as state backend – utkarsh baranwal Oct 18 '19 at 12:30
-
We are storing state as key because I am calculating median on that particular key. – utkarsh baranwal Oct 18 '19 at 12:38
-
Seems like it should work. Have you checked the taskmanager log files? – David Anderson Oct 18 '19 at 14:27
-
Hi David, after 2 days, now, I can see the sst. Is this normal or is there any issue in my setup – utkarsh baranwal Oct 20 '19 at 14:12
-
I don’t know what RocksDB does in cases where the state and update rate are small enough that the disk isn’t needed. – David Anderson Oct 22 '19 at 06:30