1

I have a Cassandra 2.1 cluster using Leveled Compaction Strategy.

Base on my calculation, the cluster will run out of space before compaction kick in automatically when it reaches next level. For that reason, I have a cron job that runs "nodetool compact" every week to run a full (major) compaction to remove tomb stoned data points.

I noticed that full compaction consumes very little CPU/network resources. With bigger data set, full compaction runs for days.

I have tried to "setcompactionthroughput" to higher number (128MB/s instead of 32MB/s by default, even tried to set it to 0 (no limit), but full compaction speed doesn't seem to change at all.

Is there anything I can tune to make it faster? Thanks in advance.

Cary Li
  • 241
  • 1
  • 5
  • 9

1 Answers1

0

There are very few cases where you should run full compaction via nodetool compact - it causes what you're likely seeing now (a single huge data file, which never naturally compacts with other sstables, even/especially when other deletions have happened).

Recovering from the state your in isn't trivial, but is possible. If you have a lot of cpu/IO to spare, you can try toggling from STCS to LCS, and LeveledCompactionStrategy will naturally split up that huge file into thousands of tiny files, and will be much more aggressive about rewriting those files over time (so tombstones are compacted away much more regularly). This is very much CPU and IO intensive, so don't do it if you're near tipping. Also, it will duplicate all data on disk for a short period, so you'll need to be under 50% disk utilization to do this.

If you're over 50% disk utilization, you've backed yourself into a corner, and you'll probably need to add more disk temporarily in order to recover.

Jeff Jirsa
  • 4,391
  • 11
  • 24
  • Thanks Jeff. I'm using LCS today. There's spare CPU and IO, I also make sure disk usage is <50% for the major compaction. Switching compaction strategy may work - but changing compaction strategy is cluster wide and almost kick off re-compaction on whole cluster, where as right now I can schedule compaction between racks (EC2 AZs). Sounds a little scary. But again thanks for pointing it out! – Cary Li Oct 03 '17 at 17:56
  • Just to be clear - you're using LCS now and doing full compaction? or you have LCS on other tables? You can roll out a compaction strategy per-node using JMX (compaction strategy is mutable through JMX using jconsole or jmxterm, so you can change it one node at a time, or one AZ at a time, if you decide you want to do it but dont want the IO hit on all nodes at the same time). You could also use a single node change via JMX to test effectiveness. – Jeff Jirsa Oct 03 '17 at 22:18
  • I'm using LCS now and doing full compaction. It's interesting learn this JMX trick. – Cary Li Oct 03 '17 at 23:18