Kafka Streams: Dynamically Configure RocksDb

Question

I want to tune performance of Kafka Streams, and for that I have to play with RocksDb configuration values.

I see I can use the StreamsConfig.ROCKSDB_CONFIG_SETTER_CLASS_CONFIG to set the configuration for RocksDB. Like shown here.

But I would like a way to configure it dynamically from configuration (in order to change the thresholds without having to compile and deploy all my code.

Is there an option to somehow give concrete implementation instance instead of a class name? (This will allow to set the thresholds, for example, using a properties file)

score 3 · Accepted Answer · answered Apr 05 '19 at 09:57

Well, You can pass the configuration parameters from the properties file at the runtime and set the ClassName in the StreamsConfig.ROCKSDB_CONFIG_SETTER_CLASS_CONFIG. This way, you won't have to recompile and redeploy your code again and again. Instead at runtime, you will be able to pass different-2 properties value.

Example : You can implement the CustomRockDBConfig as below:

public class CustomRocksDBConfig implements RocksDBConfigSetter {

    public static long blockCacheSize = 50*1024*1024L;
    public static long blockSize = 4096L;
    public static boolean cacheIndexAndFilterBlock = false;

    public static Logger log = Logger.getLogger(CustomRocksDBConfig.class);

    @Override
    public void setConfig(String storeName, Options options, Map<String, Object> configs) {

        BlockBasedTableConfig tableConfig = new BlockBasedTableConfig();
        // Reducing default block cache size
        tableConfig.setBlockCacheSize(blockCacheSize);
        // Increasing the block size as default block size is only 4KB
        tableConfig.setBlockSize(blockSize);
        // Index and Filter block
        tableConfig.setCacheIndexAndFilterBlocks(cacheIndexAndFilterBlock);

        options.setTableFormatConfig(tableConfig);
        options.setMaxWriteBufferNumber(2);


    }

}

While setting the StreamsConfig properties, add below properties.

CustomRocksDBConfig.blockCacheSize = properties.get("blockCacheSize");
CustomRocksDBConfig.blockSize = properties.get("blockSize");
CustomRocksDBConfig.cacheIndexAndFilterBlock = properties.get("cacheIndexAndFilterBlock");
properties.put(StreamsConfig.ROCKSDB_CONFIG_SETTER_CLASS_CONFIG, CustomRocksDBConfig.class);

No need to recompile the code! It will always read the values from the runtime properties file.

Kafka Streams: Dynamically Configure RocksDb

1 Answers1