Using Rsyslog sending to Loggly and scrub mongodb log data

Question

I am trying to send information to loggly via rsyslog with data from mongodb 4.4.2. However I cannot get the data in a way that I can manipulate it and scrub out certain information. When I follow the guide on the Loggly site it works for non mongodb information. If I leave %$!msg% as %msg% I get the mongodb data but I am not able to manipulate it.

config file for reading mongo logs

module(load="imfile")

#RsyslogGnuTLS
$DefaultNetstreamDriverCAFile /etc/ssl/certs/ca-bundle.crt

# Input for FILE1
input(type="imfile" tag="mongo_lou_qa" ruleset="filelog" file="/var/log/mongodb/mongod.log") #wildcard is allowed at file level only

$template LogglyFormat,"<%pri%>%protocol-version% %timestamp:::date-rfc3339% %HOSTNAME% %app-name% %procid% %msgid% [token@41058  tag=\"tag1\" tag=\"tag2\" ] %$!msg%" 

set $!msg = $msg;

if re_match($!msg,'([0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9])')    
then 
{
  set $!ext = re_extract($!msg,'([0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9][0-9])',0,1,"");
  set $!msg= replace($!msg, $!ext, "xxxxxxxxx");
}
  
ruleset(name="filelog"){
action(type="omfwd" protocol="tcp" target="logs-01.loggly.com" port="6514" template="LogglyFormat" StreamDriver="gtls" StreamDriverMode="1" StreamDriverAuthMode="x509/name" StreamDriverPermittedPeers="*.loggly.com")
}

Mongodb sample log

{"t":{"$date":"2021-01-01T00:00:00.000-00:00"},"s":"I",  "c":"ACCESS",   "id":20000,   "ctx":"conn79","msg":"Successful authentication","attr":{"mechanism":"ABC","principalName":"__system","authenticationDatabase":"local","client":"0.0.0.0:00000"}}

I dont see any string of 9 digits in your example log for the re pattern to match. Also, I understood that you needed to define the `ruleset()` before you coul use it with `input(ruleset=)`, but that may just be with older versions. — meuh, Jun 08 '21 at 15:52
@meuh Yes there are no 9 digits in the example so I would expect to see the full string returned but I'm not even seeing that. — ArbitraryDice, Jun 08 '21 at 17:42
@meuh - That solved the issue. Thanks - Is there any way I can upvote or make your answer as the answer? — ArbitraryDice, Jun 08 '21 at 19:27

score 0 · Answer 1 · answered Jun 08 '21 at 15:20

0

You may use jq to extract or manipulate data from your logfile.

Note, you can also generate syslog messages by setting

systemLog:
   destination: syslog

then you don't have to write to a file and read it again.

answered Jun 08 '21 at 15:20

Wernfried Domscheit

54,457
9
76
110

score 0 · Accepted Answer · answered Jun 08 '21 at 20:22

rulesets are used to partition the inputs so that only some rules apply to them. It seems that variables like $!msg and $.msg are local to each ruleset, so setting them outside a ruleset will not affect the values in an action that is inside a ruleset. The documentation is not very clear on just what else needs to be inside the ruleset, for example the template probably does not need to be.

Using Rsyslog sending to Loggly and scrub mongodb log data

2 Answers2