capture multiline events with rsyslog and storing them to file

Question

We have a centralized rsyslog infrastructure capturing events from TCP sent by devices around the world using imtcp module.

The idea is to read from syslog (TCP) and store the events to disk, one line per event. The events are later processed by other consumers.

As far as we can see, some events are splitted in multiple events once they are stored on the disk breaking the rest of our process.

Capturing one single package with tcpdump, we confirmed that the source syslog is sending us the whole event containing multiple lines (typical java exceptions).

[root@xx xx.xx.xx.xx]# tcpdump -i bond0 tcp port 50520 -A -c 1
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on bond0, link-type EN10MB (Ethernet), capture size 262144 bytes
12:12:26.062110 IP xx.xx.xx.xx.com.41444 > xx.xx.xx.com.50520: Flags [P.], seq 3270590174:3270590613, ack 2646946316, win 27, options [nop,nop,TS val 3937801207 ecr 2623497312], length 439
E....`@.<.ML..A....N...X..>...2......q.....
....._d`<13> xxx #2.0.#2021 02 10 12:19:50:898#+00#Info#com.xx.xx.xx.xx.xx#
##JavaEE/xx#xx#xx#JavaEE/xx#com.xx.xx.xx.xx.APIServiceHandler#xx#xx##xx#xx##0#Thread[HTTP Worker [@xx],5,Dedicated_Application_Thread]#Plain##
Is the user getting thru SSO? xx:true#

1 packet captured
44 packets received by filter
2 packets dropped by kernel

As this is a global system, we cannot request the device owners to modify the format, all the actions should take place on our side.

This is our rsyslog.conf file

$MaxMessageSize 128k
# Global configuration/modules
module(load="imtcp" MaxListeners="100")
module(load="imfile" mode="inotify")
module(load="impstats" interval="10" resetCounters="on" format="cee" ruleset="monitoring")
module(load="mmjsonparse")
module(load="mmsequence")
module(load="omelasticsearch")
module(load="omudpspoof")

# Include all conf files
$IncludeConfig /etc/rsyslog.d/*.conf

And this is our template that reads from tcp and writes to file (etc/rsyslog.d/template.conf)

template(name="outjsonfmt_device" type="list") {
         constant(value="{")
         property(outname="device_ip" name="fromhost-ip" format="jsonf")
         constant(value=",")
         property(outname="time_collect" name="timegenerated" dateFormat="rfc3339" format="jsonf")
         constant(value=",")
         constant(value="\"device_type\":\"device\"")
         constant(value=",")
         property(outname="collector_id" name="$myhostname" format="jsonf")
         constant(value=",")
         property(outname="msg" name="rawmsg-after-pri" format="jsonf" )
         constant(value="}\n")
}

template(name="device-out-filename" type="string" string="/data1/input/device/%fromhost-ip%/device_%$now-utc%_%$hour-utc%.log")
ruleset(name="writeRemoteDataToFile_device") {
         action(type="omfile" dynaFileCacheSize="10000" dirCreateMode="0700" FileCreateMode="0644" dirOwner="user" dirGroup="logstash" fileOwner="user" fileGroup="user" dynafile="device-out-filename" template="outjsonfmt_device")
}
input(type="imtcp" port="50520" ruleset="writeRemoteDataToFile_device")

How can we configure rsyslog to escape line breaks in the middle of an event, prior to write the event to disk? We already tried $EscapeControlCharactersOnReceive with no success and other similar parameters

Answer 1

The imtcp has a module parameter DisableLFDelimiter which you could try setting to on to ignore line-feed delimiters, assuming your input has an octet-count header. The page says, This mode is non-standard and will probably come with a lot of problems.

module(load="imtcp" MaxListeners="100" DisableLFDelimiter="on")