Munin combine network traffic graph from multiple servers

Question

I've been searching through munin wiki and docs, tried various combinations for setting custom graphs but all ended unsuccessfull.

Thing is that I actually cannot understand the mechanism behind creating custom graphs at all, and all I was doing was trial and error effort without understanding what actually happend. Munin docs are very poor on this matter.

I am trying to simply generate a graph with network traffic from 5 servers, so I can have quick comparisment of the traffic on each.

I am monitoring just if_eth0 on 5 servers and want to have one more graph with all 5 servers shown there.

Answer 1

In default /etc/munin/munin.conf (stable 2.0) you find this template:

# A more complex example of a host tree
#
## First our "normal" host.
# [fii.foo.com]
#       address foo
#
## Then our other host...
# [fay.foo.com]
#       address fay
#
## IPv6 host. note that the ip adress has to be in brackets
# [ip6.foo.com]
#       address [2001::1234:1]
#
## Then we want totals...
# [foo.com;Totals] #Force it into the "foo.com"-domain...
#       update no   # Turn off data-fetching for this "host".
#
#   # The graph "load1". We want to see the loads of both machines...
#   # "fii=fii.foo.com:load.load" means "label=machine:graph.field"
#       load1.graph_title Loads side by side
#       load1.graph_order fii=fii.foo.com:load.load fay=fay.foo.com:load.load
#
#   # The graph "load2". Now we want them stacked on top of each other.
#       load2.graph_title Loads on top of each other
#       load2.dummy_field.stack fii=fii.foo.com:load.load fay=fay.foo.com:load.load
#       load2.dummy_field.draw AREA # We want area instead the default LINE2.
#       load2.dummy_field.label dummy # This is needed. Silly, really.
#
#   # The graph "load3". Now we want them summarised into one field
#       load3.graph_title Loads summarised
#       load3.combined_loads.sum fii.foo.com:load.load fay.foo.com:load.load
#       load3.combined_loads.label Combined loads # Must be set, as this is
#                                                 # not a dummy field!
#
## ...and on a side note, I want them listen in another order (default is
## alphabetically)
#
# # Since [foo.com] would be interpreted as a host in the domain "com", we
# # specify that this is a domain by adding a semicolon.
# [foo.com;]
#       node_order Totals fii.foo.com fay.foo.com

And this example creates a new graph showing the load of two servers (taken from http://munin-monitoring.org/wiki/LoaningData ):

[example.org;Overview]
address 127.0.0.1 
use_node_name no
update no

loadall.update no
loadall.graph_title Load of all servers
loadall.graph_args --base 1000  
loadall.graph_vlabel Load
loadall.graph_scale no
loadall.graph_category system

loadall.graph_order \
    Server1=sv1.example.org:load.load \
    Server2=sv2.example.org:load.load

Answer 2

The first thing that is not right away apparent is that every host must belong to some group. So, this syntax is invalid:

[;myhost]
    address  ip.add.re.ss;

And this way you get a group with no hosts:

[myhost;]
    ...

You can check it by adding the following code after the following line:

foreach my $k (keys %$gah) {
    DEBUG "-- $k";
    foreach my $h (keys %{$gah->{$k}{'hosts'}}) {
        DEBUG "   $h";
    }
}

Then run (a lot of output, you might probably want to add | tail -n 100 or something):

$ su - munin -s /bin/bash -c '/usr/share/munin/munin-update --debug'
...
2019/06/14 16:31:43 -- myhost1
2019/06/14 16:31:43 -- myhost2
2019/06/14 16:31:43    myhost2
2019/06/14 16:31:43 -- myhost3
2019/06/14 16:31:43    myhost3
...

So,

[myhost]                 # group "myhost", conaining host "myhost"
[foo.com]                # group "com", containing host "foo.com"
[group;foo.com]          # group "group", containing host "foo.com"
[group1;group2;foo.com]  # group "group1", containing group "group2",
                         # which in its turn contains host "foo.com"

With that out of the way, the syntax is basically as follows:

[group1;host1]
    ...
[group1;host2]
    ...
[group1;virt_host1]
    virt_plugin1.graph_title Some title
    virt_plugin1.graph_order resulting_field_name1=host1:plugin1.field_name1 resulting_field_name2=host2:plugin2.field_name2

virt_plugin1, resulting_field_name1, resulting_field_name2 are all arbitrary names. virt_plugin1 appears in the breadcrumbs of the page. resulting_field_name* becomes a graph's label. Unless you override it:

[group1;virt_host1]
    ...
    virt_plugin1.resulting_field_name1.label = my_field_name

Regarding where to get (how to confirm) plugin and field names (data sources):

$ nc host1 4949
# munin node at host1
fetch plugin1
field_name1.value 0.17
.

Then, if you're borrowing data from a different group, you've got to specify it explicitly:

[group1;host1]
    ...
[group2;host2]
    ...
[group2;virt_host1]
    virt_plugin1.graph_title Some title
    virt_plugin1.graph_order resulting_field_name1=group1:host1.plugin1.field_name1 resulting_field_name2=host2:plugin2.field_name2

A note on when to expect changes to the config to take effect.

When html_strategy is cgi, munin-html (as part of munin-cron) writes state to /var/lib/munin/htmlconf.storable every 5 minutes. The state contains settings of every host, and is used by munin-cgi-html as a config.

So, either you wait for the next update, or trigger the update yourself:

$ su - munin -s /bin/bash -c /usr/share/munin/munin-html

Or:

$ su - munin -s /bin/bash -c munin-cron

In the same vein, munin-update updates /var/lib/munin/datafile.storable. But datafile part is marked to include base, as opposed to htmlconf one. Which means that contents of /etc/munin/munin.conf is merged into datafile part. And munin-cgi-graph uses datafile part for a config.

So you can just restart munin-cgi-graph for it to notice the changes.

And the last thing, munin-cgi-graph sets Expires header. That means you've got to somehow reset/disable browser cache for browser to make the request.

In case you notify one of them, but not the other... Let me leave it as an exercise for the reader :)