I am using Splunk for log monitoring and Nagiosxi for other monitoring-related services on Linux servers. I want to integrate Splunk with my NagiosXi server so that when Splunk receives a log status code, e.g. 500, then it forwards the error to NagiosXi and an alert is displayed in the Nagiosxi console.
Is this possible? How?
Involving Splunk might be difficult without knowledge of the API and building custom Nagios plugins. However, it would be possible to design a check (eg. via NRPE) that parses the log files for a 500 code with:
tail -n 100 log.log | grep 500
...but this runs into issues in terms of overhead and reliability. Tailing a log repeatedly hits the disk for every check. To make the check reliable you will have to extend the line count (-n 100 in my example) to make sure nothing is missed. So, the more reliable you wish the check to be, the worse it will scale.
Not to mention, this solution cuts Splunk out of the picture completely. You could similarly create queries against Splunk's API (or database), but it may take some knowledge of writing custom check scripts.
As you're already using a commercial solution-- have you considered adopting a Nagios-based solution that includes a logserver? op5 has a full package including a logger and check plugins for querying logs based on filters you set up in the GUI.
Full disclosure: I work for op5, and deal with this sort of question daily.
