1

I have setup new Cassandra 3.3 cluster. Then I use jvisualvm to monitor Cassandra read/write latency by using MBean (jmx metric).

The result of read/write latency is always stable in all nodes for many weeks whereas read/write request in that cluster have normally movement (heavy or less in some day).

As I use jvisualvm to monitor Cassandra 2.0 cluster. The read/write latency have normally behavior. It have movement depending on read/wire requests.

I wonder that Why the read/write latency statistics of Cassandra 3.0+ are always stable? And I think it is incorrect result. (I have load tested in Cassandra v3.3, v3.7).

[Updated] I have found bug relate with this issue.

Cassandra metric flat. https://issues.apache.org/jira/browse/CASSANDRA-11752

The detail show that this problem has been solved in C* version 2.2.8, 3.0.9, 3.8. But after I have tested in version 3.0.9, The result of latency still show flat line.

Any Idea?

Thanks.

tipwimon
  • 11
  • 3

1 Answers1

0

have not found any metrics problem When using C*3.3

first,try to monitor with jconsole,have met same issue？

second,which attribute do you see?avg value or percentage?there value always count from node up,so it is common to see percentage value is same.but not always happens on average value.try to restart cassandra node and check the value.

Code Stone
  • 129
  • 3
  • 12
  • Yes, I have met the same issue when using jconsole to monitor the latency. – tipwimon Jul 07 '16 at 07:29
  • The metric that met the issue is org.apache.cassandra.metrics.ClientRequest.Latency.Write.p99 org.apache.cassandra.metrics.Table.WriteLatency.all.p99 org.apache.cassandra.metrics.ClientRequest.Latency.Read.p99 org.apache.cassandra.metrics.Table.ReadLatency.all.p99 **All percentile of the read and write latency** for ClientRequest and Table does not have any movement. – tipwimon Jul 07 '16 at 07:35
  • it is common that 99th percentile keep stable. – Code Stone Jul 07 '16 at 08:33
  • But 50percentile, 75percentile and all others are stable all week long. Whereas read/write request is heavy or less depending on application load. Is it possible? – tipwimon Jul 07 '16 at 10:51
  • can you restart the node,and add workload,then attach the value. – Code Stone Jul 08 '16 at 01:09
  • Yes. The result is same as I said. I have used cassandra-stress to add word load many time and I have used graph option of cassandra-stress to report the result and also use jconsole to monitor it. Result of **all percentile** does not have any thing change. It show the same value until the end of stress test. Are you see normal latency result on Cassandra 3.3, 3.7? Am I do something wrong? – tipwimon Jul 11 '16 at 09:02
  • see normal latency resutl on Cassandra 3.3.The result is same as you said,how much is it?10000+ microsecond? and what is value of attribute "count",Does it increased？ – Code Stone Jul 12 '16 at 01:36
  • Yes, count is increasing. Including m1_rate, m5_rate and all other rate are have normally result. It have movement follow by request count. Latency 2.82 microsecond for the request count 51.97k. – tipwimon Jul 12 '16 at 10:43