Template to store Rsyslog logs with MongoDB

Question

I am trying to insert the logs coming from RSyslog to a MongoDB database.

The logs stored into MongoDB have to respect the following structure :

{
    "_id" : ObjectId("55b8c845a671d907a0ab9e0b"),
    "receptionTime" : "2015-06-12 14:29:45",
    "reportedTime" : "2015-06-12 14:29:45",
    "priority" : "6",
    "facility" : "23",
    "host" : "uacm3-3a-fscr01",
    "service" : "apacheaccess",
    "message" : "My messsage",
    "syslogTag" : "apache-access-fscr:"
}

According to Rsyslog documentation (http://www.rsyslog.com/doc/v8-stable/configuration/templates.html#standard-template-for-writing-to-files) , I have designed the following template :

template(name="BSON" type="list") {
    constant(value="\"receptionTime\": \"")
    property(name="timegenerated")
    constant(value="\", \"reportedTime\": \"")
    property(name="timereported")
    constant(value="\", \"priority\": \"")
    property(name="syslogseverity")
    constant(value="\", \"facility\": \"")
    property(name="syslogfacility")
    constant(value="\", \"host\": \"")
    property(name="hostname")
    constant(value="\", \"service\": \"")
    property(name="programname")
    constant(value="\", \"message\": \"")
    property(name="msg")
    constant(value="\", \"syslogTag\": \"")
    property(name="syslogtag")
    constant(value="\"")
    }

Unfortunately, the logs stored in MongoDB do not respect the required structure at all. Here is what is stored :

{
    "_id" : ObjectId("55e715b25ea0c0a9fbbf8b0f"),
    "timegenerated" : "Sep  2 17:28:50",
    "timereported" : "Sep  2 15:27:57",
    "syslogseverity" : "5",
    "syslogfacility" : "21",
    "hostname" : "uacm3-3b-acd01",
    "programname" : "Sep",
    "msg" : "Some message",
    "syslogtag" : "Sep"
}

Do you have any idea about what I am doing wrong ?

Is the date format the only problem? Because it's weird that field names are different, you hardcoded them. And having date as programname/syslogtag - can you post the message as written via the RSYSLOG_ForwardFormat template to a file? For timestamp, try adding dateFormat="rfc3339". You'd need to use position.from and position.to to get the bits you need, like property(name="timereported" dateFormat="rfc3339" position.from="1" position.to="10") # gives the date constant(value=" ") # insert the space property(name="timereported" dateFormat="rfc3339" position.from="12" position.to="19") # rest — Radu Gheorghe, Sep 03 '15 at 05:00
@RaduGheorghe Thanks for your answer. Unfortunately, the real problem is related to the names of the properties. Event if they are hardcoded, the result obtained is completely different as you can see. For the date thing, I aware of that, it could be corrected thanks to a formating function. — Félix Veysseyre, Sep 03 '15 at 07:37
This is really weird. It's either a strange bug in the template system or the wrong template is used. Can you post the entire rsyslog.conf? — Radu Gheorghe, Sep 04 '15 at 07:15

score 1 · Accepted Answer · answered Sep 04 '15 at 08:04

I found a solution, but I still do not understand why the former method was not working:

template(name="BSON" type="list") {
    property(name="timegenerated" outname="receptionTime")
    property(name="timereported" outname="reportedTime")
    property(name="syslogseverity" outname="priority")
    property(name="syslogfacility" outname="facility")
    property(name="hostname" outname="host")
    property(name="programname" outname="service")
    property(name="msg" outname="message")
    property(name="syslogtag" outname="syslogTag")
    }

Template to store Rsyslog logs with MongoDB

1 Answers1