NRPE: "CHECK_NRPE: Error receiving data from daemon." error in the gui, but the check works from terminal, what could be the cause?

Question

I'm monitoring a few Linux servers using NRPE and most of the checks are working. Actually, the only check which doesn't work is check_disk. Running a remote check_disk from the Nagios server terminal,:

[root@nagios]# /usr/lib64/nagios/plugins/check_nrpe -H 10.200.X.X -c check_disk -a '-w 20% -c 10% /'
DISK OK - free space: / 271971 MB (97% inode=99%);| /=8321MB;236233;265762;0;295292

Running a local check_disk from the monitored server's terminal:

[root@Monitored ~]# /usr/lib64/nagios/plugins/check_nrpe -H 127.0.0.1 -c check_disk -a '-w 20% -c 10% /'
DISK OK - free space: / 271971 MB (97% inode=99%);| /=8321MB;236233;265762;0;295292

This check_disk command returns error "CHECK_NRPE: Error receiving data from daemon" on each server it checks, which leads me to believe it's a problem with the way the service or the command are written, so here they are: The command from the /etc/nagios/nrpe.cfg file:

[root@Monitored ~]# grep disk /etc/nagios/nrpe.cfg 
command[check_local_disk]=sudo /usr/lib64/nagios/plugins/check_disk -w $ARG1$ -c $ARG2$ -p $ARG3$

The command from the /etc/nagios/commands.cfg file:

# 'check_local_disk' command definition
define command{
        command_name    check_local_disk
        command_line    $USER1$/check_disk -w $ARG1$ -c $ARG2$ -p $ARG3$
        }

The service from services.cfg file:

define service{
   servicegroups        Basic Functionality
   hostgroup_name       db_hosts,vm_hosts,linux_hosts
   host_name            localhost
   service_description  Check free disk space /
   check_command        check_nrpe!check_local_disk!20%!10%!/
   use                  generic-service
}

Here's the command definition of check_nrpe:

define command{
        command_name    check_nrpe
        command_line    /usr/lib64/nagios/plugins/check_nrpe -H $HOSTADDRESS$ -c $ARG1$
}

Here's the information regarding NRPE from /var/log/messages on the monitored server:

Jun 10 12:57:01 virt2 nrpe[755]: INFO: SSL/TLS initialized. All network traffic will be encrypted.
Jun 10 12:57:01 virt2 nrpe[756]: Starting up daemon
Jun 10 12:57:01 virt2 nrpe[756]: Server listening on 0.0.0.0 port 5666.
Jun 10 12:57:01 virt2 nrpe[756]: Server listening on :: port 5666.
Jun 10 12:57:01 virt2 nrpe[756]: Warning: Daemon is configured to accept command arguments from clients!
Jun 10 12:57:01 virt2 nrpe[756]: Listening for connections on port 0
Jun 10 12:57:01 virt2 nrpe[756]: Allowing connections from: 127.0.0.1,10.200.X.X

Do you have any idea how to solve this issue? Thanks in advance

The `check_local_disk` command isn't relevant, as that'll only run on the Nagios server itself. Can we see how the service is defined in your Nagios config? — Flup, Jun 10 '14 at 10:18
Can we see the definition of `check_nrpe` in your `commands.cfg`? — Flup, Jun 10 '14 at 10:31
Also can you check the logs on the monitored server for anything written by `nrpe`? Location varies by distro, but grep for it in `/var/log`. — Flup, Jun 10 '14 at 10:32

MadHatter · Accepted Answer · 2014-06-10T13:41:21.590

The data you show us are self-inconsistent.

On nagios, you show yourself invoking the check on server monitored by using check_nrpe like so:

[root@nagios]# /usr/lib64/nagios/plugins/check_nrpe -H 10.200.X.X -c check_disk -a '-w 20% -c 10% /'

but when you show us monitored's nrpe.cfg file, the check is defined by a different name:

command[check_local_disk]=sudo /usr/lib64/nagios/plugins/check_disk -w $ARG1$ -c $ARG2$ -p $ARG3$

(I apologise for the lack of formatting, but I wanted the ability to highlight more than I wanted a monospaced font). It seems to me that the order of the parameters is also wrong, but I'm no expert on nrpe configurations that permit arguments to be passed.

At any rate, the -c foo argument to check_nrpe must match the command[foo] in the nrpe.cfg, and it doesn't. That can only mean that either something you've showed us isn't so, or you're proving that you can invoke check_nrpe against the wrong server.

Edit: I think I've already been fairly clear about trying to solve it. You say the GUI doesn't run the check correctly. OK. So you're right that the normal next step is to run it from the command line, but it's quite important to run the same check against the same client. You've shown us the client's nrpe.cfg, so assuming that the client really is 10.200.X.X, from the server show us the results of

[root@nagios]# /usr/lib64/nagios/plugins/check_nrpe -H 10.200.X.X -c check_local_disk -a '-w 20% -c 10% /'

It would be useful to see the check_nrpe entry from the server's commands.cfg file, as well, just to confirm that all lines up. We will now be trying to do what the server's doing, so if the command above fails, we can debug and fix the failure. If it succeeds, we have to drill down a little harder.

Hi MadHatter, If you will look on the "Command Definition", you will see that the command name is "check_local_disk" and the command line is actually "check_disk" and not "check_local_disk", that means that while using the `-c foo`, the correct command name to use is "check_disk" - which is the command I used. More than that, the only problem happens in the GUI itself, if i run the checks manually they work properly... — Itai Ganot, Jun 10 '14 at 12:10
For commands executed via `nrpe`, the only server-side lookup of `commands.cfg` is to find the definition of `check_nrpe`. The **argument** to check_nrpe (`-c foo`) is used to invoke whatever plugin's `command[]` argument matches that name in the remote client's `nrpe.cfg`. I repeat that your setup is self-inconsistent, which may well be the root cause of the problem(s). Despite your assurances that when you "*run the checks manually they work properly*", I am by no means sure that this is the case. — MadHatter, Jun 10 '14 at 12:21

NRPE: "CHECK_NRPE: Error receiving data from daemon." error in the gui, but the check works from terminal, what could be the cause?

1 Answers1