3

I am doing rocksdb compaction using compactfiles api. I checked the compaction stats, the read and write is very slow.. I am seeing the read write and write rate to be around 10MBps. which is very low.. Because my flushes are happening at rate around 330MBps.. I have set the compaction readahead size to 1MB in rocksdb options... I have set the block cache to 10GB.. I am compacting 32 files of each size 32 MB. Do need to set any other options to increase the compaction speed?

Following are the logs containing the compression stats.

compacted to: files[1 0 0 0 0 0 0] max score 52.38, MB/sec: 10.9 rd, 10.9 wr, level 0, files in(0, 32) out(1) MB in(0.0, 980.9) out(980.9), read-write-amplify(0.0) write-amplify(0.0) OK, records in: 986848, records dropped: 0 output_compression: NoCompression

(Original Log Time 2023/07/28-06:31:17.610339) EVENT_LOG_v1 {"time_micros": 1690525877610328, "job": 0, "event": "compaction_finished", "compaction_time_micros": 94778569, "compaction_time_cpu_micros": 0, "output_level": 0, "num_output_files": 1, "total_output_size": 1028547792, "num_input_records": 986848, "num_output_records": 986848, "num_subcompactions": 1, "output_compression": "NoCompression", "num_single_delete_mismatches": 0, "num_single_delete_fallthrough": 0, "lsm_state": [1, 0, 0, 0, 0, 0, 0]}

suresh
  • 4,084
  • 10
  • 44
  • 59

1 Answers1

0

Shared your question with the Speedb Hive - the most active RocksDB/Speedb/LevelDB/KVS community and this is what we have for you:

Several parameters control the speed of the compaction and are usually set to max speed. you can use sub compactions which will use multiple threads. I feel the compaction read-ahead is too large (depending on the HW). Anyway to understand the cause you can look at CPU usage and disk usage during the compaction

You can find the Speedb hive here and (once you've registered) the link to the thread with your question here, if you have more questions or need additional info

Dan Carfas
  • 46
  • 2