1

I'm using Zabbix's zabbix_sender.sh script to push exception stack traces to my remote Zabbix monitoring server.

zabbix_sender.sh requires a key and a value for any data that it sends. It can read data from stdin, but this overrides any key variable specified. Because my stdin data isn't formatted like Zabbix expects, I need to pass in "value" as an argument. Hopefully this provides some context.

What I want to accomplish is capturing the multiline result from grep into a variable, preserving newlines, so that I can call the zabbix_sender.sh script with that variable as an argument.

What I've tried so far looks like this:

tail -Fn0 /var/log/uwsgi.log | grep "Exception:" -A 100 | (read tback; /usr/local/zabbix/bin/zabbix_sender -z myzserver.com -s MyHostName -k uwsgi_traceback -o $tback)

As best I can tell, that never invokes zabbix_sender.sh.

To test, I've tried using this command, which doesn't seem to work either:

tail -Fn0 /var/log/uwsgi.log | grep "Exception:" -A 100 | (read errorlines; echo "$errorlines" > /tmp/errorlines.txt)

The /tmp/errorlines.txt file is never created.

How can I capture grep's output lines into a variable so that I can call another script with that variable as an argument?

skyler
  • 465
  • 3
  • 8
  • 17
  • Why don't you use [log file monitoring](https://www.zabbix.com/documentation/2.0/manual/config/items/itemtypes/log_items)? – Michael Hampton Apr 05 '14 at 17:09
  • Zabbix's built in log file monitoring is great at alerting me that something went wrong, but I haven't managed to get it to record any surrounding lines. Even using the `output` argument to `log`, I can't seem t make it capture more than the line that triggered it. If you know of a way to make this work, please share it with me. – skyler Apr 05 '14 at 17:53

3 Answers3

2

Here my approach.

  1. Use zabbix "log" type key to monitor error pattern in a log-file (/var/log/uwsgi.log is the case here).
  2. invoke zabbix remote command triggered by the above 1. This remote command fetch lines surrounding the error by linux command tail(1).

The advantages of this approach are:

  1. No need to install&setup agent-host side special script (like zabbix_sender.sh discussed above). zabbix 'log' type item has been already provided such a kind of purpose.
  2. no zabbix_sender.sh-like script on agent-host mean that there is no additional CPU & memory consumption. Fetching several lines surounding error in the log happens only at trigger.

Let me explain the detail how to setup below:

  1. register regular expression in zabbix from Administration > General > Regular expressions > [New regular expression] like "error|fail|fatal". Let's assume the variable name is @uwsgi_error_pattern.
  2. register item from Configuration > Hosts > [Target host] line > Items > [Create Item] with the following attributes:
    • Description: [any name]
    • Type: Zabbix agent(active)
    • Key: log[/var/log/uwsgi.log,@uwsgi_error_pattern]
    • Type of information: Log
  3. register 2nd item from Configuration > Hosts > [Target host] line > Items > [Create Item] with the following attributes, to accept fetched log flagment sent by zabbix remote command(mentioned later):
    • Description: [any name]
    • Type: Zabbix trapper
    • Key: my_app.fetch_uwsgi_log
    • Type of information: Text
  4. register trigger from Configuration > Hosts > [Target host] line > Triggers > [Create trigger] with the following attributes:
    • Name: uwsgi log monitor on {HOSTNAME}
    • Expression ({[Target host]:log[/var/log/uwsgi.log,@uwsgi_error_pattern].iregexp(@uwsgi_error_pattern)})#0&({[Target host]:log[/var/log/uwsgi.log,@uwsgi_error_pattern].nodata(300)})=0
  5. register action which remotely executes to fetch data surrounding the error line of the log file as follows:
    • Name: Fetch latest uwsgi log on error
    • Action Operations:
      • Operation Type: Remote command
      • Remote command: {HOSTNAME}:zabbix_sender -z [zabbix-server] -s {HOSTNAME} -k my_app.fetch_uwsgi_log -o "`tail -200 /var/log/uwsgi.log`"

NOTE-1: key name "my_app.fetch_uwsgi_log" in the above step is just example. We can define any unique name to bind zabbix_sender and the item.

NOTE-2: You may need AllowRoot=1 on /etc/zabbix/zabbix_agentd.conf to allow zabbix-agent read uwsgi.log.

Fumisky Wells
  • 121
  • 5
1

You must turn off word splitting in bash, eg, by clearing IFS:

export IFS=""
set NEWVAR=`your tail|grep expression`

Now echo $NEWVAR has the newlines.

mkaama
  • 191
  • 1
  • 5
  • This will run once and capture tail's output, no? Will it follow the file as it's written to by the app that I'm logging? – skyler Apr 05 '14 at 17:57
  • Unfortunately that didn't work `export IFS="" tail -Fn0 /var/log/uwsgi.log | grep "Exception:" -A 100 | (read errorlines; echo "$errorlines" > /tmp/errorlines.txt)` – skyler Apr 05 '14 at 18:02
  • I marked up this answer because I didn't know about the IFS trick for capturing multiline output into a variable. – pdwalker Jan 14 '15 at 01:42
1

The problem is the tail part; since it is in continuos mode it wont ever spit out something for "read" to read.

This should work:

#!/bin/bash
echo "0" >/tmp/numberoflines
IFS=''
while [ 1 ]
do
    NUMBER=$(cat /tmp/numberoflines)
    LINES=$(wc -l < /var/log/uwsgi.log)
    DIFFERENCE=$(($LINES-$NUMBER))

    if [ $DIFFERENCE != 0 ]; then
            exception=$(tail -n $DIFFERENCE /var/log/uwsgi.log | grep "Exception:" -A 100)
           /zabbix/bin/zabbix_sender -z myzserver.com -s MyHostName -k uwsgi_traceback -o $tback) $exception;
    fi

sleep 5;
echo "$LINES" >/tmp/numberoflines
done
metacom
  • 306
  • 1
  • 6