Cubism and metrics in Elasticsearch (statsd/diamond)

Question

In a test environment setup, I'm using diamond to send server stats and metrics to statsd and then using statsd's elasticsearch backend to send them to elasticsearch.

The elasticsearch data looks like this:

{
  "hits" : {
    "hits" : [ {
      "_source":{"ns":"servers","grp":"server1","tgt":"vmstat","act":"pswpout","val":"0","@timestamp":"1438565055000"}
    }, {
      "_source":{"ns":"servers","grp":"server1","tgt":"vmstat","act":"pgpgin","val":"0","@timestamp":"1438565055000"}
    }, {
      "_source":{"ns":"servers","grp":"server1","tgt":"vmstat","act":"pswpin","val":"0","@timestamp":"1438565055000"}
    }, {
      "_source":{"ns":"servers","grp":"server1","tgt":"cpu","act":"total.nice","val":"0","@timestamp":"1438565055000"}
    }, {
      "_source":{"ns":"servers","grp":"server1","tgt":"cpu","act":"total.irq","val":"0","@timestamp":"1438565055000"}
    }, {
      "_source":{"ns":"servers","grp":"server1","tgt":"cpu","act":"total.guest","val":"0","@timestamp":"1438565055000"}
    }, {
      "_source":{"ns":"servers","grp":"server1","tgt":"diskspace","act":"_logs.byte_used","val":"209944576","@timestamp":"1438565055000"}
    }, {
      "_source":{"ns":"servers","grp":"server1","tgt":"diskspace","act":"_logs.byte_free","val":"887513440256","@timestamp":"1438565055000"}
    }, {
      "_source":{"ns":"servers","grp":"server1","tgt":"diskspace","act":"_logs.byte_avail","val":"842419666944","@timestamp":"1438565055000"}
    }, {
      "_source":{"ns":"servers","grp":"server1","tgt":"diskspace","act":"_logs.inodes_used","val":"11","@timestamp":"1438565055000"}
    } ]
  }
}

How would I go about having this data rendered using cubism? I can extract the data using elasticsearch.js, but just generally stumped about how I would go about getting this rendered, so that, I can visualize the time-series data for each of the different actions, for instance.

Does it have to be cubism or you need functional charts for these metrics? — Sergei Rodionov, Aug 25 '15 at 17:02
@SergeiRodionov looking for functional charts using these metrics. — adele dazim, Aug 26 '15 at 05:00

score 1 · Accepted Answer · answered Aug 21 '15 at 10:14

1

You can try https://github.com/bernd/statsd-influxdb-backend or http://graphite.wikidot.com backend. For those backend http://grafana.org/ provides great visualisation tools.

If you choose Graphite cubism actually supports it too as a data provider according to their website: https://square.github.io/cubism/

answered Aug 21 '15 at 10:14

Andrew Andrew

302
1
2

To expand on this answer: You don't want to use ES because it is not a time series database and you don't want to use Kibana because it can only chart 1 metric per query (charting min, max, and mean on one chart requires 3 queries instead of 1 query with a complex result set). Use grafana and influxdb and don't fight a stack's defaults. – chugadie Oct 04 '15 at 18:00

score 0 · Answer 2 · answered Aug 26 '15 at 06:22

Take a look at ATSD time chart. If this is functional enough, Axibase Time-Series Database Community Edition is free. By the way, it supports Graphite wire protocol directly so you can just stream data from your diamond daemons directly into the database.

Disclosure: I work for this company.

Cubism and metrics in Elasticsearch (statsd/diamond)

2 Answers2