1

I have a log file (zope/plone event.log) which using custom string (e.g "-----") as divider between events, how grok pattern for parsing this log file to logstash should be?

This is an example how the log look like:

------
2014-07-21T12:13:30 INFO ZServer HTTP server started at Mon Jul 21 12:13:30 2014
        Hostname: localhost
        Port: 8401
------
2014-07-21T12:13:44 WARNING SecurityInfo Conflicting security declarations for "setText"
------
2014-07-21T12:13:44 WARNING SecurityInfo Class "ATTopic" had conflicting security declarations
------
2014-07-21T12:13:47 INFO DocFinderTab Applied patch version 1.0.5.
Dinh Anh Cuong Nguyen
  • 43
  • 4

1 Answers1

0

You should start with the multiline codec or filter to create a single event for processing.

EDIT:

The doc gives this example:

filter {
  multiline {
    pattern => "pattern, a regexp"
    negate => boolean
    what => "previous" or "next"
  }
}

And describes what 'negate' and 'what' do. Hopefully 'pattern' make sense.

So, how about "every line that doesn't start with a date belongs with the prior line"? That might be something like this:

filter {
  multiline {
    negate => 'true'
    pattern => "^%{TIMESTAMP_ISO8601} "
    what => 'previous'
  }
}

You'd be left with the "----" at the end of each line. Since you don't need them as delimiters, you can get rid of them (before the multiline filter stanza):

if message =~ /^-+$/ {
    drop{}
}
Alain Collins
  • 16,268
  • 2
  • 32
  • 55
  • I've checked the multiline filter here http://logstash.net/docs/1.4.1/filters/multiline. However there is only 1 way to specify multiline event is define the pattern of the next line or previous line. However, in my case, the event is divided by "-----" string, not the end line, so the beginning pattern of every could be vary – Dinh Anh Cuong Nguyen Sep 10 '15 at 16:41
  • You can say that "everything that is not a ----- belongs to the previous line", etc. – Alain Collins Sep 10 '15 at 18:09
  • Yes, it's true, but I'm new with grok and I dont know how to write that rule. Do you know how to do it or point me a place that I can learn how to do it? Thanks in advanced! – Dinh Anh Cuong Nguyen Sep 11 '15 at 08:39