Extract values from MESSAGE in Syslog-NG

Question

I receive syslog messages as follows:

[nms321@uhp.1.3.6.1.4.1.8000.10 net_id="325" station="431" fault="16384"] [NotificationHandler] NMS error: STCRC/NOTTS Net: Enas Station: CED Elmas serial-num: 11347

Without Python parser, I correctly receive logs.

Now, I need to extract some values from the above MESSAGE and I configured syslog-ng.conf as follows:

source s_network {
        udp();
        tcp();
};

python {

import re

class SngRegexParser(object):

    def init(self, options):

        pattern = options["regex"]
        self.regex = re.compile(pattern)
        self.counter = 0
        return True

    def deinit(self):

        pass

    def parse(self, log_message):

        decoded_msg = log_message['MESSAGE'].decode('utf-8')
        match = self.regex.match(decoded_msg)
        if match:
            for key, value in match.groupdict().items():
                log_message[key] = value
            log_message['MY_COUNTER'] = str(self.counter)
            self.counter += 1
            return True
        return False
};

parser my_python_parser{
    python(
        class("SngRegexParser")
        options("regex", "station: (?P<station>\\d+), error: (?P<error>\\d+), Net: (?P<Net>\\d+), Station: (?P<Station>\\d+), serial-num: (?P<serial-num>\\d+) (?P<padding>.*$)")
    );
};

destination d_file { file("/var/log/app_network.log"); };

destination d_mysql {
        sql(
                type(mysql)
                host ("127.0.0.1")
                username("USER")
                password("PASSWORD")
                database("syslog")
                table("logs")
                columns("host", "facility", "priority", "datetime", "program", "msg", "StNum", “Error”, “Network”, “StName”, “Serial”)
                values("${HOST}", "${FACILITY_NUM}", "${LEVEL_NUM}", "${R_YEAR}-${R_MONTH}-${R_DAY} ${R_HOUR}:${R_MIN}:${R_SEC}", "${PROGRAM}", "${MSGONLY}", "${station}", "${error}", "${Net}", "${Station}", "${serial-num}")

                indexes("host", "program", "datetime", "facility", "priority")
                null("")
        );
};


log { source(s_local); source(s_network); parser(my_python_parser); destination(d_file); destination(d_mysql); };

But after reloading of Syslog-NG, I don't receive any log and Syslog-NG status shows this error:

Error initializing new configuration, using the old config

What is wrong?

Thank you guys

score 0 · Answer 1 · edited Oct 07 '21 at 08:14

There are multiple issues here.

Resolving the first one makes it possible to start syslog-ng with the new configuration.

Group names of capture groups must be valid Python identifiers:

Exception while calling a Python function; caller='my_python_parser', class='SngRegexParser', function='init', exception='error: bad character in group name 'serial-num' at position 112' ``` Fix: Use (?P<serial_num>\\d+) instead of (?P<serial-num>\\d+)

The regexp does not match on the provided message. Adjust it accordingly.

The beginning of the message looks like structured-data from RFC 5424. syslog-ng has native/fast parsers for this format, have a look at the syslog-procotol flag of network().
If you shared your full config, not just a snippet:

Do not forget to add the config version (@version: 3.19) to the top, and define the s_local source.

Thank you for your reply, I continued my post with the next answer. — Dspi, Mar 15 '19 at 15:27
What do you mean by "not working"? If you don't want to drop unmatching messages, just do not return with `False`. — MrAnno, Mar 15 '19 at 16:21

score 0 · Answer 2 · answered Mar 15 '19 at 15:26

thank you for your reply.

1- I did this operation yestarday.

2- I did this operation yesterday, now I'm using:

@version: 3.19

source s_network {
        syslog(transport(udp) port(514));
        syslog(transport(tcp) port(514));
};

3- modified.

Unfortunately, the configuration with the correction is not working.

Now the configuration file is as follows:

source s_network {
        syslog(transport(udp) port(514));
        syslog(transport(tcp) port(514));
};

python {

import re

class SngRegexParser(object):

    def init(self, options):

        pattern = options["regex"]
        self.regex = re.compile(pattern)
        self.counter = 0
        return True

    def deinit(self):

        pass

    def parse(self, log_message):

        decoded_msg = log_message['MESSAGE'].decode('utf-8')
        match = self.regex.match(decoded_msg)
        if match:
            for key, value in match.groupdict().items():
                log_message[key] = value
            log_message['MY_COUNTER'] = str(self.counter)
            self.counter += 1
            return True
        return False
};

parser my_python_parser{
    python(
        class("SngRegexParser")
        options("regex", "station: (?P<station>\\d+), error: (?P<error>\\d+), Net: (?P<Net>\\d+), Station: (?P<Station>\\d+) (?P<padding>.*$)")
    );
};

destination d_file { file("/var/log/app_network.log" template("error: $error Net: $Net Station: $Station my_counter: $MY_COUNTER\n")); };


log { source(s_network); parser(my_python_parser); destination(d_file); };

Any other idea?

Extract values from MESSAGE in Syslog-NG

2 Answers2