I have a Kafka streams application, where each instance handles ~30 partitions and has 1 state store (resulting in a total of 30 RocksDB instances). Assuming the application can scale in/out, the total number of RocksDB instances can change. I'd like to set limits on the total amount of memory allocated for the block cache and memtable sizes, regardless of the actual number of instances running. I've extended the RocksDBConfigSetter class (the pod's memory limit is 2GB):
class RocksDbConfig extends RocksDBConfigSetter {
import RocksDbConfig._
override def setConfig(storeName: String,
options: Options,
configs: util.Map[String, AnyRef]): Unit = {
val tableConfig: BlockBasedTableConfig = options
.tableFormatConfig() match { case config: BlockBasedTableConfig => config }
// Block cache and memtable settings
tableConfig.setBlockCache(cache)
// Increase max block size to make index block smaller
tableConfig.setBlockSize(MAX_BLOCK_SIZE)
// Index and filter blocks are allocated off-cache and can grow unbounded
// setting them in block cache will guarantee bounded memory usage
tableConfig.setCacheIndexAndFilterBlocks(true)
tableConfig.setPinTopLevelIndexAndFilter(true)
tableConfig.setCacheIndexAndFilterBlocksWithHighPriority(true)
options.setWriteBufferManager(writeBufferManager)
options.setMaxWriteBufferNumber(NUM_OF_MEM_TABLES)
options.setWriteBufferSize(MAX_WRITE_BUFFER_SIZE)
}
override def close(storeName: String, options: Options): Unit = {}
}
object RocksDbConfig {
def toMB(size: Int) = size * 1024 * 1024L
def tokB(size: Int) = size * 1024L
val INDEX_FILTER_BLOCK_RATIO = 0.5
val NUM_OF_MEM_TABLES = 2
val TOTAL_BLOCK_CACHE_MEMORY = toMB(700)
val MAX_WRITE_BUFFER_SIZE = toMB(64)
val TOTAL_MEMTABLE_MEMORY = NUM_OF_MEM_TABLES * MAX_WRITE_BUFFER_SIZE
val TOTAL_OFF_HEAP_MEMORY = TOTAL_BLOCK_CACHE_MEMORY + TOTAL_MEMTABLE_MEMORY
val MAX_BLOCK_SIZE = tokB(8)
// All objects must be created in the companion object,
// to make sure all RocksDB instances get the same object.
val cache = new LRUCache(TOTAL_OFF_HEAP_MEMORY, -1, false, INDEX_FILTER_BLOCK_RATIO)
val writeBufferManager = new WriteBufferManager(TOTAL_MEMTABLE_MEMORY, cache)
}
But cannot tell wether these limits are enforced separately on each instance, or on all of them. According to Confluent's wiki, there's a way to enforce these limits on all instances:
As of 5.3.0, the memory usage across all instances can be bounded, limiting the total off-heap memory of your Kafka Streams application. To do so you must configure RocksDB to cache the index and filter blocks in the block cache, limit the memtable memory through a shared WriteBufferManager and count its memory against the block cache, and then pass the same Cache object to each instance
Update: I've added a log to setConfig() and it was printed 30 times, so I guess it means each instance got it's own copy of the config
My questions are:
- Does it mean that setting
NUM_OF_MEM_TABLESto2will result in a total of two memtables on all instances or 2 * num_of_instances? - How do I pass the same Cache object to each instance (as stated in the quoted text above)?
