2

Currently I am debugging performance issue with Apache Cassandra. when Memtable for the column family is filled, it is queued to be flushed to SSTable. This flushing happens often when you perform massive writes. When this queue is filled up, writes are blocked until next successful completion of flush. This indicates that your node cannot handle writes it is receiving.

Is there a matrix in nodetool indicating this behaviour? In other words, I want a data indicating a node cannot keep up with writes it is receiving.

Thanks!!

user_1357
  • 7,766
  • 13
  • 63
  • 106

3 Answers3

4

Thats not really true for a couple years. The active memtable is switched and a new memtable takes its position as live. New mutations occur on this live memtable while the "to be flushed" memtables are included in local reads. The MemtableFlushWriter thread pool has the flush tasks queued on it. So you can see how many are pending there (under tpstats). The mutations backing up you can also see under the MutationStage.

Ultimately

nodetool tpstats

Is likely what your looking for.

Chris Lohfink
  • 16,150
  • 1
  • 29
  • 38
3

I want a data indicating a node cannot keep up with writes it is receiving.

Your issue is likely bound to disk I/O not being able to handle the throughput --> flushes of memtables queue up --> writes are blocked

the command dstat is your friend to investigate I/O issues. Some others linux commands may be also handy. Read this excellent blog post from Amy Tobey: https://tobert.github.io/pages/als-cassandra-21-tuning-guide.html

Is there a matrix in nodetool indicating this behaviour?

nodetool tpstats

Aaron
  • 55,518
  • 11
  • 116
  • 132
doanduyhai
  • 8,712
  • 27
  • 26
2

I believe you're looking for tp (thread pool) stats. 

nodetool tpstats

Typically blocked FlushWriters indicates that your storage system is having trouble keeping up with the write workload. Are you using spinning disks by chance? You'll also want to keep an eye on iostat in this case as well.

Here's the docs for tpstats: https://docs.datastax.com/en/cassandra/2.1/cassandra/tools/toolsTPstats.html

MarcintheCloud
  • 1,643
  • 9
  • 9