0

I have application logging events in JSON format for later structured queries, etc. Now I have a task to log the same messages in plain text (CSV probably) and I really don't want to log each event twice in both formats.

Is it possible to convert within rsyslog ? I have an option of tracing the output file and piping it to converter and then log with different tag, but that seems suboptimal.

Any other ideas ?

Thanks !

stimur
  • 894
  • 6
  • 11

1 Answers1

1

You might want to have a look at mmjsonparse. It appears to do the sort of thing you want to do. You'll need rsyslog 6.6 or higher (7 or higher is recommended). The sample config found here explains it quite well:

# load needed modules
module(load="imuxsock") # provides support for local system logging
module(load="imklog") # provides kernel logging support
module(load="mmjsonparse") #for parsing CEE-enhanced syslog messages

# try to parse structured logs
action(type="mmjsonparse")

# define a template to print field "foo"
template(name="justFoo" type="list") {
    property(name="$!foo")
    constant(value="\n") #we'll separate logs with a newline
}

# and now let's write the contents of field "foo" in a file
action(type="omfile"
    template="justFoo"
    file="/var/log/foo")

You'll still need to use logger or some module of whatever language your application is written in which interacts with rsyslog to write the messages...

# logger '@cee: {"foo":"bar"}'
# cat /var/log/foo
bar

If you send an unstructured log, or invalid JSON, nothing will be added to the log.

drew
  • 173
  • 1
  • 6
  • thanks ! this is great for predefined set of fields. Is there way to extract values only from dynamic set of fields ? – stimur Sep 14 '15 at 19:16
  • Probably best to introduce some form of structure so that the template knows what to extract. Log messages should be relatively consistent if possible. You may want to look at projects like [Lumberjack](https://fedorahosted.org/lumberjack/) which try define a best-practice event syntax for JSON formatted log entries. But if your fields differ considerably from each other, I can imagine that the resultant config could become quite considerable! – drew Sep 14 '15 at 19:23
  • this is exactly the reason I'm using JSON, to store logs in ELK :). I just wanted rsyslog to catch up when format is changed, but reconfiguring field set is fine too I think. Thank you ! – stimur Sep 14 '15 at 19:24