I'm using cassandra 2.0, and I have created a column family that looks like this:
CREATE TABLE user_id_timestamp_index (
user_id int,
timestamp text,
PRIMARY KEY (user_id, timestamp)
) WITH
bloom_filter_fp_chance=0.010000 AND
caching='KEYS_ONLY' AND
comment='' AND
dclocal_read_repair_chance=0.000000 AND
gc_grace_seconds=864000 AND
index_interval=128 AND
read_repair_chance=0.100000 AND
replicate_on_write='true' AND
populate_io_cache_on_flush='false' AND
default_time_to_live=0 AND
speculative_retry='NONE' AND
memtable_flush_period_in_ms=0 AND
compaction={'class': 'LeveledCompactionStrategy'} AND
compression={'sstable_compression': 'LZ4Compressor'};
I've written over 2 million rows to this table without any issues, and I perform numerous deletes as well.
The problem arises after about 10k deletes or so in rapid succession, and I start encountering numerous rpc_timeouts. A simple "delete from user_id_timestamp_index where user_id = 5 AND timestamp = '12345'" via cqlsh fails during this period.
Things I've noticed and tried:
- During rpc timeouts, Load Average on 2 nodes (out of 5) shoot up to about 50.
- Compaction is done almost every 5 minutes during these load-intensive writes and deletes.
- During rpc_timeouts, tpstats shows pending mutation stages: MutationStage 64 (active) 395 (pending) 48182373 (completed) 0 0
- Timeouts tend to occur when memtable data size exceeds 3 mb for this CF.
- After I perform a nodetool flush, pending mutations goes to zero and rpc time disappears, until memtable size creeps up again to past 3 mb.
My question is, there a configuration I can tune? For example, is the solution to simply force a memtable flush on this column family every 5 minutes? Lessen the write load on this table? A way to have faster writes and reduce pending stages? Or is there a better solution?
