How to determine logical/physical reads and writes in cassandra?

Question

I'am currently investigating read and writes operations on cassandra. I would like to know about the read and writes that cassandra performs on the memtable and sstable. What is the closest value of the cassandra metrics? All I found are latency metrics, but this isn't what I'm searching for.

I would rather like to know how many real reads and writes occurs to build an abstract cost model for some specific operations.

Thank you!

Edit:

So it looks like ReadLatency and WriteLatency (Table Metrics) are what I've been searching for. However, if I want to read the metrics the Read Count is always 0.

My setup:

PopOS! 20.04 / Lenovo P1 G2 (512GB nvme SSD, 16GB Ram)
Cassandra 3.11.7 (Docker Container)
Datastax 4.5.1 Java Driver
Gathering metrics through Nodetools (been using the executable inside of the docker container) or with Telegraf 1.15.2 + Jolokina 1.6.2

My Telegraf config:

  [[inputs.jolokia2_agent.metric]]
    name  = "TableRead"
    mbean = "org.apache.cassandra.metrics:keyspace=*,name=ReadLatency,scope=*,type=Table"
    tag_keys = ["keyspace", "name", "scope"]
    field_prefix = "$2_"

  [[inputs.jolokia2_agent.metric]]
    name  = "TableWrite"
    mbean = "org.apache.cassandra.metrics:keyspace=*,name=WriteLatency,scope=*,type=Table"
    tag_keys = ["keyspace", "name", "scope"]
    field_prefix = "$2_"

Both are showing the Read Count as Zero on my Table (Write Counter is incrementing as expected) when running a query through my Java Application like this:

ResultSet rs = cqlSession.execute("SELECT * from myTable");
for(Row r : rs) {
    int id = rs.getObject("someColumn");
}

Why is the Read counter not incrementing?

I had a look at `nodetool tablestats` [datastax documentation](https://docs.datastax.com/en/ddac/doc/datastax_enterprise/tools/nodetool/toolsTablestats.html) and they're reporting (local) read/write count. However there is no way to access that values through the metrics they're offering. Do I miss something? [metrics documentation](https://cassandra.apache.org/doc/latest/operating/metrics.html) — Sebastian, Aug 01 '20 at 09:04
It's a little misleading, but looks like `Timer` object reports both latency & request rate. You might be able to get read and write requests/s from the `CoordinatorReadLatency` and `CoordinatorWriteLatency` metrics, respectively — ShaharZ, Aug 04 '20 at 05:34

score 1 · Answer 1 · answered Aug 04 '20 at 06:48

1

There are at least 2 types of the read/write metrics in the Cassandra:

local read/writes - the corresponding operations executed by replicas
client level read/writes - how this happens on the coordinator level. One read/write at coordinator level is converted into N local read/writes determined by consistency level and replication factor

Cassandra's Metrics page provides a list of available metrics, and you can get metrics specific for a table, or aggregated for a coordinator - you need to look to Latency metrics, like, ReadLatency for table, or ClientRequests.Read for coordinator.

For reads there are also other important metrics, like, number of SSTables hit during the read of specific table (the more you hit, worse is the latency), number of tombstones, etc. On the host level - things like, key cache hit rate is also very important. DataStax has a separate page about "important metrics" as part of their best practices docs.

answered Aug 04 '20 at 06:48

Alex Ott

80,552
8
87
132

Thank you so much! After looking in-depth I think ReadLatency and WriteLatency on a per table level is what I was looking for. However, I am not seeing any Reads. How is a read defined in that context? Even if I loop through a whole table (`select * from table` in Java, and then iterating over the ResultSet) there are no reads.. Thank you! – Sebastian Aug 06 '20 at 15:08
How do you collect data? – Alex Ott Aug 06 '20 at 15:44
Not sure if I understood your question correctly.. I have a single-node setup (just for research purpose) on which I am creating some random data - after that I do some schema manipulation and try to understand the behavior. I can really see perfectly the writes when I'm running an update, but no reads at all. Using the latest Datastax Java Driver. – Sebastian Aug 06 '20 at 16:02
I meant - how do you get these metrics - via JMX with JConsole, or with Prometheus, etc.? – Alex Ott Aug 06 '20 at 16:12
I'm using nodetools or JMX (with Telegraf, InfluxDB and Grafana). But both show 0 reads. Could be probably related to this: https://stackoverflow.com/questions/34577401/cassandra-nodetool-cfstats-read-count-is-always-0 – Sebastian Aug 06 '20 at 16:16
Please add more information to Q about your setup - Cassandra version, etc. – Alex Ott Aug 06 '20 at 16:41
Edited it into my main question. Appreciate your help! – Sebastian Aug 06 '20 at 18:11

How to determine logical/physical reads and writes in cassandra?

1 Answers1