Six months ago I made note to the pattern for referring to stacked graph data sources in munin:
load.double.stack one=localhost.localdomain:load.load two=localhost.localdomain:load.loadThis syntax evaluates to:
graph.value.stack line=host.domain:plugin.value
I’ve been using multigraph more since then, which is a boon to performance, but it complicates stacked graphs a little. This hurts because it remains very difficult to tell why your graphs are not drawing when you incorrectly reference a data source. To debug, as the munin user (use ‘su -l munin’, ‘sudo -s -u munin’ or ‘chpst -u munin’) run:
/usr/share/munin/munin-graph --service 'load.double.stack' --debug
Be sure to replace “load.double.stack” with the name of the graph you’re trying to draw.
The munin wiki example for stacked graphs explains data source names as:
snmp_ups_current.inputtotal.sum \ ---------------- ---------- --- | | | | | `-- The sum mechanism | `--------- One of this virtual plugin's values `----------------------- The name of the virtual plugin ups-5a:snmp_ups_ups-5a_current.inputcurrent \ ups-5b:snmp_ups_ups-5b_current.inputcurrent ------ ----------------------- ------------ | | | | | `------ The "inputcurrent" value from the real plugin | `------------------------ The real plugin's name (symlink) `---------------------------------------- The host name from which to seek information
However, with multigraph the name of the plugins symlink isn’t necessarily the name of the graph. The trick I found was to connect the the munin node and call the multigraph plugin, looking for the ‘multigraph’ line.
$ nc localhost 4949 # munin node at server.example.org cap multigraph # tell munin-node that you are multigraph aware cap multigraph fetch diskstats # fetch the diskstats multigraph plugin multigraph diskstats_latency sdb_avgwait.value 0 multigraph diskstats_latency.sdb avgwait.value 0 .
I’ve removed a significant portion of the returned data here. Pay attention to the fact that this plugin returned a “diskstats_latency” graph that contains data for all of the disks, as well as individual graphs for each disk, here “diskstats_latency.sdb” In this example your stacked graph configuration would be:
disk.double.stack \ one=localhost.localdomain:diskstats_latency.sdb.avgwait \ two=localhost.localdomain:diskstats_latency.sdb.avgwait -1- ----------2---------- -----------3--------- ---4---
(1) The alias and label for this host or data point
(2) The configured node name of the host
(3) The original graphs name, either the plugin or multigraph name
(4) The value from the plugin/graph
Notice that while the period is used to separate the value from the rest of the field, there may be periods in the rest of the field. Also keep in mind that in the past I have seen dashes in configured graph names become underscores at the end of the day.
Thanks for all the munin posts, Bryan. Every now and then I am tempted to replace munin with graphite entirely and start collecting system stats in graphite. But these posts have helped me stay on munin for a little while longer 🙂