Graphite metrics shows wrong results

Question

I'm using Graphite with Codahale to record metrics from my java server. I have a block code that looks something like that:

public void foo() {
  try (Timer.Context ignored = myTimer.start()) {
    // Some code
  }
}

When I look at today's event count (each timer is also a counter) I see that we're around the hundred of hit counts a minute, which means a a few thousands an hour. When I widen the date range to include yesterday as well, I see that the results are in the millions range and I could not figure out why.

The results are shown after nonNegativeDerivative operation on the metric

Today's results:

With yesterday's results:

Graphs above do not look like counter graphs. Counters are usually monotonically increasing and you need to apply nonNegativeDerivative() function in Graphite to get an exact number of calls per minute (per your metric interval). — deniszh, Mar 07 '17 at 14:50
You're right, that's after applying nonNegatvieDerivative(). Editing. — Avi, Mar 07 '17 at 15:00
Then I suspect what your problem is. You're should apply sum of derivatives (**sumSeries(nonNegativeDerivative(...))**) and not derivative of sums - http://www.jilles.net/perma/2013/08/22/how-to-do-graphite-derivatives-correctly/ — deniszh, Mar 07 '17 at 15:04

score 3 · Answer 1 · answered Mar 07 '17 at 15:06

If using nonNegativeDerivative() - please apply derivative first, and then sumSeries() - not vise versa. Please check http://www.jilles.net/perma/2013/08/22/how-to-do-graphite-derivatives-correctly/

Also, you need to set up correct aggregation (sum) for counters like described in http://obfuscurity.com/2012/05/A-Precautionary-Tale-for-Graphite-Users

score 2 · Answer 2 · answered Apr 05 '17 at 17:46

you do not want to use sum on dropwizard counters. You want to use "last" they are incrementing decrementing gauges. You also want perSecond() and hitcount(10s) instead of the derivative.

http://graphite.readthedocs.io/en/latest/functions.html#graphite.render.functions.perSecond

Graphite metrics shows wrong results

2 Answers2