nagios wrongly reports packet loss

Question

Lately, on my nagios 3.2.3 install (CentOS5, monitoring ~ 300 hosts, 1150 services) has sdtarted to occasionally report high packet loss on 50-60 hosts at a time. Problem is it's bogus. Manual runs of ping (or its own check_ping binary) finds no fault with any of the affected hosts. The only possible cures I found so far are:

1. run all the checks manually (they will succeed but it may act up again on next check)
2. acknowledge and wait for the problem to go away (may take several ours)

I suspect (but have no particular reason other than single rescheduled checks succeeding) that the problem may lay with all the checks being mass scheduled together - in which case introducing some jitter in the scheduling (how?) might help. Or it may be something completely different.

Ideas, anyone?

Edit:

For people interested in constructive debate (rather than point scoring). I am not trying to measure packet loss. Network performance is not my concern in this instance, and if it was, it would be investigated with the proper tools for the job. NAGIOS (for the unwary) is mostly used to check upness in host servicesand to generate alerts. When it starts generating large amounts of fishy alerts is therefore highly annoying. I am 99.9% positive that the problem is either due to either:

1. some Nagios/Nagios-Plugin snag
2. some system (memory-cpu- i/O - network stack) problem

possibly caused by the burst of requests sent by the nagios scheduler. The packet losses are all above 50% - if they were real, our phones would be melting. So far I have no evidence for (2), so I am looking for "prior art" in (1). I may well be mistaken in my belief, but, if I have to reach for wireshark or similar, a suggestion on what to look for would be greatly appreciated.

Answer 1

After you have verified the packet loss by different tools, First of all you need to find out which plugin is actually checking for packet loss. Locate that plugin and manually run it after the interval defined in the nagios, and check its output if that can give you a clue. The problem doesnt seems to be that packet loss is there but its the fault plugin. once you have verified the plugin output, then compare that output with output of other tools (to see if it shows any packet loss and if others dont). Usually the plugin is check_ping.

Answer 2

I had a similar problem on my first try with Nagios. While trying to solve the problem, I found this blog post, which states that the problem may occur if you are trying to ping on a IPV6 server without an IPV6 address on your server.

So the solution is to rewrite the "check_ping" command on your Nagios configuration object files. On one of our .cfg files, I added the following:

define command {
    command_name    check_ping_ipv4
    command_line    $USER1$/check_ping -4 -H $HOSTADDRESS$ -w 3000.0,80% -c 5000.0,100% -p 5
}

Please note the "-4" parameter after the check_ping command. It will force the ping to use IPV4 only. After defining the command above, I could use it in a service definition. For example:

define service {
    service_description     PING
    host_name               MYHOST
    check_command           check_ping_ipv4!100.0,20%!500.0,60%
    use                     generic-service
}

Answer 3

Manual runs of ping (or its own check_ping binary) finds no fault with any of the affected hosts

That's a really dumb way to check for packet loss. You should be comparing the retransmits logged for the NIC at intervals (netstat -r) or capturing the traffic using a tool like pastmon or wireshark. Since:

1) you've already said that the packet loss occurs in bursts - how do you know you were running a ping on a path during the time packet loss was occurring?

2) small amounts of packet loss can have a big impact on throughput - which is why we monitor them - if you want to confirm packet loss of 1%, then you'd need to send at least 200 packets across the path - how many did you send?

3) However the overriding WTF here is that TCP, and to a lesser extent UDP, behaves very differently from ICMP - the latter is far less affected by congestion issues (even assuming a consistent 1500 MTU)

i.e. you've provided no valid evidence that the packet loss is bogus. You have however provided evidence that you don't really understand what you were trying to measure.

yet other hosts on the same networks do not have the same loss

Do you think packet loss only occurs between hosts? This is way wrong.