0

I'm looking at monitoring and alerting for some of our business metrics in a web app (pageviews, signups, etc). We already use Nagios and Munin for a wide variety of server monitoring and alerting cases, which is why I've started there.

I could write custom plugins for Nagios that calculate our statistics / control charts and check for when these metrics dip below desirable levels (Warning and Critical), but I'd also like to know when these metrics spike above the expected levels (lots more signups - we did something right!).

Is there a way to create custom alert levels in Nagios or Munin to accomodate these positive alerts, or is there another tool I should be looking at to solve for this case? The ideal tool would:

  1. Include more alert levels (Critical, Warning, OK, Improved, Spiking)
  2. Allow me to see additional data about the report that generated the alert (expected value of the metric and observed value)
  3. (Nice to have) allow me to graph the history of the metric so I can visualize the observations after receiving the alert
Travis
  • 333
  • 1
  • 3
  • 11

2 Answers2

1

I use nagios for alerting and ganglia for metrics gathering. Often times my tool that sends data into ganglia (via gmetric) also sends alerts into nagios (via send_nsca, aka passive alerts.) If you write your tool properly it can detect changes from the last period to the current period. This way you can, for example, detect changes in certain metrics. Is the average response time at this time X% higher/lower than the last time? Did the queue size increase/decrease X% more/less?

I used munin for a while, but it didn't have the granularity of ganglia. Also, creating a new metric in ganglia is as easy as calling gmetric. BAM, you have a new graph. I also find it useful to have the last deploy time displayed on our metrics graphs (just a vertical line showing the time of the deploy.) That way you can better visualize what's going on and when.

toppledwagon
  • 4,245
  • 25
  • 15
1

Many NAGIOS plugins already do exactly what you want, and allow WARNING and CRITICAL thresholds to be specified as a range, alerting when the value is outside (or, optionally, inside) that range. For example, from /usr/lib/nagios/plugins/check_procs -h:

[...]
Usage:
check_procs -w <range> -c <range> [-m metric] [-s state] [-p ppid]
 [-u user] [-r rss] [-z vsz] [-P %cpu] [-a argument-array]
 [-C command] [-t timeout] [-v]
[...]
 -w, --warning=RANGE
   Generate warning state if metric is outside this range
 -c, --critical=RANGE
   Generate critical state if metric is outside this range
[...]
RANGEs are specified 'min:max' or 'min:' or ':max' (or 'max'). If
specified 'max:min', a warning status will be generated if the
count is inside the specified range
[...]

So this plugin already has the idea that "X is problematic if it's more than C or less than B, and it's an impending disaster if it's less than A or more than D" (for A<B<C<D). That sounds to me very much like what you asked for, and no rearchitecting of NAGIOS (into which the OK/WARN/CRIT concept is pretty heavily embedded) is needed.

MadHatter
  • 79,770
  • 20
  • 184
  • 232
  • This sounds like it could work - my only real qualm with it is that I can't tell from the alert whether it's Warning: something bad happened, or Warning: something good happened. Is there a way to make that distinction or provide details about why the alert was generated? – Travis Feb 16 '12 at 13:09
  • 1
    The alert can always provide information fed back by the plugin about why it was raised, yes. If your plugin were to return "WARNING: signups on host www3 is 12 (outside 100:1000)" or "WARNING: signups on host www3 is 12485 (outside 100:1000)" you'd get that as part of a properly-configured alert, and could act accordingly. – MadHatter Feb 16 '12 at 13:15