0

I am converting a rsyslog template to syslog-ng and I cannot find in the syslog-ng docs how to embed regex's in a template. The incoming message body looks like this:

123 1.2.3.4 4.3.2.1:80 someone@somewhere.com US

The original rsyslog template is:

$template graylog_json,"{\"version\":\"1.1\", \"host\":\"%HOSTNAME:::json%\", \"short_message\":\"Mail Authentication Log\", \"_LogDateTime\":\"%timereported:::date-rfc3339,json%\", \"_Cluster\":\"c25\", \"_ResponseCode\":\"%msg:R,ERE,1,BLANK:^[^ ]*? ([0-9]{3}) --end:json%\", \"_SourceIP\":\"%msg:R,ERE,2,BLANK:^ ([0-9]{3}) ([0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3})--end:json%\", \"_DestinationIP\":\"%msg:R,ERE,1,BLANK: ([0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}):[0-9]{2,4}--end:json%\", \"_DestinationPort\":\"%msg:R,ERE,1,BLANK: [0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}.[0-9]{1,3}:([0-9]{2,4})--end:json%\", \"_UserAccount\":\"%msg:R,ERE,1,BLANK::[0-9]{2,4} ([^ ]{1,})--end:json%\", \"_Country\":\"%msg:R,ERE,2,BLANK::[0-9]{2,4} ([^ ]{1,})( [A-Z?]{2})?--end:json%\"}\n"

The regex bits in the template parse out the relevant fields in the original message. I can't just dump messages to graylog because we use custom fields. I believe I want to use a template in syslog-ng, but I can't find examples, or even docs, showing how to embed regex's inside a template.

Rob
  • 1

1 Answers1

0

looking at the body of your message, you have the following options:

  • Parse the message with a csv-parser, using the whitespace as separator character. Note that the csv-parser will not split the IP:port, but you can run another csv-parser on the address (this time with : as separator) to do that. You can find examples for that in the syslog-ng documentation
  • Alternatively, you can write a custom syslog-ng parser in Python to process this message, and use the standard python string functions to separate the message into words and split the IP:port pair.

Using the csv-parser is probably easier and has better performance. Also, syslog-ng version 3.13 includes a graylog destination (that's not included in the docs yet, but you can find an example in this blog post Graylog as destination in syslog-ng)

Robert Fekete
  • 557
  • 3
  • 5
  • Interesting, yes, the csv-parser will do the job. I did look at the blog post, but I am posting specific custom field names in my json document. Would I frame the json message in the template() part of the csv-parser(), then pass the message to a destination(graylog2()) in the log() section? – Rob Jan 17 '18 at 15:29
  • Hi, you can use the format-json template function (https://www.balabit.com/documents/syslog-ng-ose-latest-guides/en/syslog-ng-ose-guide-admin/html/reference-template-functions.html) to format the fields of the message into json. – Robert Fekete Jan 18 '18 at 15:12