2

I want to centralize logging on my servers using syslog-ng which will write a JSON-formatted line to a file, which in turn will be picked up by logstash, which will forward it to elasticsearch. This setup works, except for some specific JSON issues.

I format in syslog-ng the log to be in JSON via a destination stanza:

destination d_json { file("/var/log/all_syslog_in_json.log" perm(0666) template("{\"@timestamp\": \"$ISODATE\", \"facility\": \"$FACILITY\", \"priority\": \"$PRIORITY\", \"level\": \"$LEVEL\", \"tag\": \"$TAG\", \"host\": \"$HOST\", \"program\": \"$PROGRAM\", \"message\": \"$MSG\"}\n")); };

This usually works fine, but sometimes the JSON ends up malformed due for instance to existing quotes in $MSG.

Is there a better way to format the message? I was looking at the built-in json-parser but it looks like it requires key-value pairs as the input, while I would like to explode the available fields into a JSON entry

EDIT & SOLUTION:

I found on Dustin Oprea' blog the exact solution:

destination d_json { file("/tmp/test.json" template("$(format-json --scope selected_macros --scope nv_pairs)\n")); };
WoJ
  • 3,607
  • 9
  • 49
  • 79

1 Answers1

-1

since you want to format the message as JSON, not parse it, you need the format-json() function of syslog-ng (see Administrator Guide > template and rewrite > Customize message format > template functions > format-json).

Recent versions of syslog-ng can send messages directly to Elasticsearch (see Administrator Guide > Destinations > Elasticsearch).

HTH, Robert

Robert Fekete
  • 552
  • 1
  • 3
  • 6
  • Following your answer, I read again the docs and while they are really, really obscure I manage to get what I needed. I updated my question accordingly. Thanks. – WoJ Oct 23 '15 at 10:54
  • Hi, which part of the docs did you find obscure? I'd be happy to improve them. – Robert Fekete Oct 26 '15 at 08:21
  • Hello Robert: when searching on how to format JSON output, I found `format-json` and `json-parser`. I was not clear on how to use either of them. The macros part does not either explicitly state which elements from the input syslog data get parsed into what fields. I was too harsh, sorry, with the "really, really obscure" statement - it is rather that for someone who is not an expert in the field it is not obvious what to use. I believe that adding a practical example and possibly explain more down-to-earth elements like `selected-macros` or `nv-pairs`would help. – WoJ Oct 27 '15 at 14:44
  • (cont'd) I also thought that a JSON **input** was required for `format-json` to be used - you mention in the example that * the source is a JSON encoded log message* - which misled me to think that it works with JSON as a source only (which is not true, after very carefully reading the docs). And nevertheless THANK YOU for this great piece of software. – WoJ Oct 27 '15 at 14:46
  • Hi WoJ, thanks a lot for your feedback and the detailed description of your problem. You were not harsh at all :) I added this to my todo list, and hope to review and improve these parts of the guide in the next weeks. If you encounter any similar problems, let me know, I really appreciate comments. – Robert Fekete Oct 29 '15 at 19:58
  • Beware of the "directly send to Elasticsearch" part: it may be incompatible with the specific version you're using and may emit cryptic errors and crash. – grin Jun 20 '18 at 22:07
  • Both links are dead. – problemofficer - n.f. Monica Oct 11 '20 at 06:14