Stackdriver-agent didn't collect monitoring data (HitRate for KeyCache)

Question

I'm configuring Stackdriver-agent in GCE VM to monitor Cassandra metrics. (based on GCP guide: https://cloud.google.com/monitoring/agent/plugins/cassandra)

I used the default setting of the link above, and they work fine. However, one metric I added doesn't work with the following error.

I tried gauge or counter for Type and Value or Count for Attribute. However, either of them doesn't work well.

Any suggestion, please.

Error

Feb 19 23:14:08 pgxxxxxxx1 collectd[16917]: write_gcm: Server response (CollectdTimeseriesRequest) contains errors: { "payloadErrors": [ { "index": 161, "valueErrors": [ { "error": { "code": 3, "message": "Unsupported collectd id: plugin: \"cassandra\" type: \"gauge\" type_instance: \"cache_key_cache-hitrate\"" } } ] } ] }

Config (added KeyCache-Hitrate metrics to the original config in the guide)

Connection part:

<Connection>
# When using non-standard Cassandra configurations, replace the below with
#ServiceURL "service:jmx:rmi:///jndi/rmi://CASSANDRA_HOST:CASSANDRA_PORT/jmxrmi"
ServiceURL "service:jmx:rmi:///jndi/rmi://localhost:7199/jmxrmi"
InstancePrefix "cassandra"
User "cassandra"
Password "xxxxxxxx"
Collect "cassandra_storageservice-load"
Collect "cassandra_Cache_KeyCache-Hits"
Collect "cassandra_Cache_KeyCache-HitRate"   <===== Added line
...
Collect "cassandra_DroppedMessage_MUTATION-Dropped"
Collect "cassandra_DroppedMessage_READ-Dropped"

MBean part:

<MBean "cassandra_Cache_KeyCache-HitRate">
    ObjectName "org.apache.cassandra.metrics:type=Cache,scope=KeyCache,name=HitRate"
    <Value>
        Type "gauge"
        InstancePrefix "cache_key_cache-hitrate"
        Table false
        Attribute "Value"
    </Value>
</MBean>

My environment stackdriver-agent 5.5.2-379.sdl.stretch cassandra 3.11.1

With sjk (Swiss Java Knife) tool, I got the following result ant it seems to me that the Attribute should be Value. $ java -jar sjk-0.9.2.jar mx -s localhost:7199 -b "org.apache.cassandra.metrics:name=HitRate,scope=KeyCache,type=Cache" -mi org.apache.cassandra.metrics:type=Cache,scope=KeyCache,name=HitRate org.apache.cassandra.metrics.CassandraMetricsRegistry$JmxGauge - Information on the management interface of the MBean (A) Value : java.lang.Object - Attribute exposed for management (O) objectName() : javax.management.ObjectName - Operation exposed for management — simotunes, Feb 19 '18 at 15:28
It seems many metrics are not supported at this moment. I tries with other metrics and they also failed with the same errors. Should try the other solution like datadog or prometheus... ref) https://cloud.google.com/monitoring/api/metrics_agent#agent-cassandra — simotunes, Feb 21 '18 at 04:32

simotunes · Accepted Answer · 2018-02-22T04:17:40.593

Following the guide for custom metrics and , I could solve my issue.

create custom metrics Follow the guide here: https://cloud.google.com/monitoring/custom-metrics/creating-metrics#monitoring-create-metric-python (From Custom metric name until call the create method. Timeseries is not required)

Also need to be authorized to access to monitoring. (Follow IAM guide).

configure cassandra plugin (.conf file) Guide here: https://cloud.google.com/monitoring/agent/custom-metrics-agent (From the top until Load the new configuration)

sample code of mine

code to create custom metrics client_request_read-latency-1minrate.py

from google.cloud import monitoring

client = monitoring.Client() descriptor = client.metric_descriptor( 'custom.googleapis.com/cassandra/client_request/latency/1minrate', metric_kind=monitoring.MetricKind.GAUGE, value_type=monitoring.ValueType.DOUBLE, labels=[monitoring.label.LabelDescriptor("operation", description="The storage operation name.")], description='Cassandra read latency rate for 1 minitue', display_name='Read latency 1 minutes rate') descriptor.create()

cassandra plugin example (following 2-1 and 2-2 in the same config file) 2-1. cassandra plugin example part 1 in

 <MBean "cassandra_custom_ClientRequest_Read-Latency">
     ObjectName "org.apache.cassandra.metrics:type=ClientRequest,scope=Read,name=Latency"
     <Value>
         Type "gauge"
         InstancePrefix "client_request_read-latency-1minrate"
         Table false
         Attribute "OneMinuteRate"
     </Value>
 </MBean>

<Connection>
    # When using non-standard Cassandra configurations, replace the below with
    #ServiceURL "service:jmx:rmi:///jndi/rmi://CASSANDRA_HOST:CASSANDRA_PORT/jmxrmi"
    ServiceURL "service:jmx:rmi:///jndi/rmi://localhost:7199/jmxrmi"
    InstancePrefix "cassandra_custom"
    User "cassandra user name"
    Password "your password"

    Collect "cassandra_custom_ClientRequest_Read-Latency"
</Connection>

2-2. cassandra plugin example part 2

<Chain "GenericJMX_cassandra_custom">
    <Rule "rewrite_genericjmx_to_cassandra_custom">
        <Match regex>
            Plugin "^GenericJMX$"
            PluginInstance "^cassandra_custom.*$"
        </Match>
        <Target "set">
            MetaData "stackdriver_metric_type" "custom.googleapis.com/cassandra/client_request/latency/1minrate"
            MetaData "label:operation" "%{plugin_instance}"
        </Target>
        <Target "replace">
            MetaData "label:operation" "cassandra_custom_" ""
        </Target>
    </Rule>
    <Rule "go_back">
        Target "return"
    </Rule>
</Chain>

<Chain "PreCache">
    <Rule "jump_to_GenericJMX_cassandra_custom">
        <Target "jump">
            Chain "GenericJMX_cassandra_custom"
        </Target>
    </Rule>
</Chain>
PreCacheChain "PreCache"

Official guide of Stackdriver monitoring manual is not easy to read and understand. I hope this will help..

Stackdriver-agent didn't collect monitoring data (HitRate for KeyCache)

1 Answers1