Logstash -- delimit event in log4net.log which may contain multiple lines

Question

Here is a typical log file generated from log4net

So, this log file is read by the logstash file input plugin.

By default, the delimiter in configuration is \n, which means each line is an event.

But in the log file above, you can see there could be multiple lines for one event. (like ERROR or FAULT or others)

How to configure Logstash to delimit the event correctly?

I suppose I could configure multiple delimiters like \nINFO \nDEBUG \nERROR \nFAULT . But the document says there can only be one delimiter.

score 4 · Accepted Answer · answered Jul 28 '15 at 14:29

The following config should delimit your events properly.

Input config:

input { 
    file {
        path => "/absolute/path/here.log"
        type => "log4net"
        codec => multiline {
                    pattern => "^(DEBUG|WARN|ERROR|INFO|FATAL)"
                    negate => true
                    what => previous
                }
      }
}

score 3 · Answer 2 · edited Apr 22 '17 at 14:23

3

What you have there is a multiline event. There is a codec that will help you process that.

The basic idea is to define a pattern that identifies the beginning of a log entry (in your case, the log level), and then roll all other lines into the previous one.

edited Apr 22 '17 at 14:23

Bhargav Rao

50,140
28
121
140

answered Jul 28 '15 at 13:51

Alain Collins

16,268
2
32
55

Right answer. I've just added an example config. – hurb Jul 28 '15 at 14:32
Someone edited this answer as it seems the multiline filter was removed in LS5. Since I know lots of people are running older versions, I'll just add that option here as a comment. – Alain Collins Apr 24 '17 at 15:58

Logstash -- delimit event in log4net.log which may contain multiple lines

2 Answers2