0

An Icinga2 plugin (written by myself) returns performance data with metrics named /var(avail_MB), /var(total_MB) and similar. Data is forwarded to an InfluxDB with Grafana as Frontend.

I'm using "GROUP BY" "tag(metric)" and "ALIAS BY" "$tag_metric" in a dashboard's panel query. The metric names are displayed correctly below the graph then.

However when I try to override series by specifying "alias or regex" /var(avail_MB) it does not seem to work, and when going back from panel configuration to dashboard, I get an error message saying "Panel rendering error '/var(avail_MB)' is not a valid regular expression.". I tried to put a backslash in front of ( and ), but that didn't help. To make matters worse, the whole graph disappeared, and when trying to open the "Query Inspector", the frontend seems to take forever (Query never appears).

What is the problem, and how could I fix it? I'm new to Icinga2, Grafana and InfluxDB (I'm just a "user" not administrator of those).

Grafana Panel Configuration before Series Override

Grafana Panel Configuration after Series Override

The color change is not applied to the graph.

Here is an example of plugin output:

OK: /var: 3114/5632MB (55.30%), slope is NaN|/var(total_MB)=5631.56MB;;;0 /var(avail_pct)=55.30%;25;5;0;100 /var(avail_MB)=3114.12MB;10;5;0;5632 /var(est_avail_MB)=nanMB;10;5;0;5632

(The "nanMB" was a bug in the plugin that has been fixed already, but that data wasn't from the machine in question.)

U. Windl
  • 3,480
  • 26
  • 54

1 Answers1

0

The problems seems to be the beginning of the string ("/var"). Grafana seems to treat every string starting with / as regular expression, and it expects any regular expression to start with /, too (it seems).

So the fix was to add a trailing /, and escape the literal / as \/.

Unfortunately this only removes the error message, but doesn't make the override work (match). It is also required to backslash-escape the parentheses and the slash(es):

Instead of /var(total_MB) you need to write /\/var\(total_MB\).

The original problem has two origins:

  1. The monitoring plugin specification at https://www.monitoring-plugins.org/doc/guidelines.html#AEN201 states: "2. label can contain any characters except the equals sign or single quote (')") states that any character except = and ' is allowed as metric name.

  2. Grafana v6.7.3 proposes the incorrect (i.e.: unescaped) values for "alias or regex".

That is how I had created the problem.

U. Windl
  • 3,480
  • 26
  • 54