Logstash - FileBeat multiline setting groups too much messages

Question

I've configured FileBeat to send multiline logs using the following config:

-
  paths:
    - /opt/wls/domains/filename.log
  input_type: log
  document_type: log_doc
  multiline:
    pattern: ^%{TIMESTAMP_ISO8601}
    negate: true
    match: after

As I know, it should append all the lines to the previous one, until it finds a line, which starts with a timestamp (TIMESTAMP_ISO8601).

In my case with this setting filebeat sends several log messages grouped to one single. My log messages start like this:

2016-10-14 20:31:07,447 INFO [ACTIVE] ExecuteThread: '5' for queue: 'weblogic.kernel.Default (self-tuning)' ...

It should match to ^%{TIMESTAMP_ISO8601}, so what can be the problem? Why are they sent as one message?

Thank You.

P.S. I've also tried with ^%{YYYY} and ^%{YEAR}patterns, but the result was the same...

All the examples in the doc have the regex pattern in single quotes. Also, a quick search shows no examples that use the named patterns from logstash in Filebeat. Try a regular regexp. — Alain Collins, Oct 16 '16 at 20:12
Are you sure that filebeat can use the logstash grok pattern? I haven't see anything about it in the doc — baudsp, Oct 17 '16 at 07:58
You'll have to write a regex, perhaps using https://github.com/logstash-plugins/logstash-patterns-core/blob/master/patterns/grok-patterns — baudsp, Oct 17 '16 at 07:59

score 1 · Accepted Answer · answered Oct 17 '16 at 11:15

As written in the comments, FileBeat doesn't support grok patterns. I wrote a regexp instead of the grok pattern, and it worked well. The supported regexps can be found here: https://www.elastic.co/guide/en/beats/filebeat/1.2/regexp-support.html and some multiline examples and tips in case of FileBeat can be found here: https://www.elastic.co/guide/en/beats/filebeat/1.2/multiline-examples.html

score 0 · Answer 2 · answered Apr 13 '18 at 02:27

Obviously GROK will not help and have to use Regex. What I did was as below and it works for me,

filebeat.prospectors:

    - type: log

      enabled: true

      paths:
          - /xxx/server.log*

      multiline.pattern: '^[0-9]{4}-[0-9]{2}-[0-9]{2}'
      multiline.negate: true
      multiline.match: after

Here I'm just looking for time stamp at the beginning of the line.

Logstash - FileBeat multiline setting groups too much messages

2 Answers2