Filebeat multiline tag is not getting added for multiline event

Question

What I have done:

I have enabled multiline pattern in filebeat.yml file:

  multiline.pattern: '^[[:space:]]+|^Caused by:'
  multiline.negate: false
  multiline.match: after

My sample log file contains multiline exceptions:

Exception in thread "main" java.lang.IllegalStateException: A book has a null property
       at com.example.myproject.Author.getBookIds(Author.java:38)
       at com.example.myproject.Bootstrap.main(Bootstrap.java:14)
Caused by: java.lang.NullPointerException
       at com.example.myproject.Book.getId(Book.java:22)
       at com.example.myproject.Author.getBookIds(Author.java:35)
       ... 1 more
Exception in thread "main" java.lang.NullPointerException
        at com.example.myproject.Book.getTitle(Book.java:16)
        at com.example.myproject.Author.getBookTitles(Author.java:25)
        at com.example.myproject.Bootstrap.main(Bootstrap.java:14)

From filebeat logs I can see that these multiline exceptions are getting parsed properly:

2017-05-15T08:21:36-07:00 DBG  Publish: {
  "@timestamp": "2017-05-15T15:21:31.768Z",
  "beat": {
    "hostname": "WIN-UV5OA3SO3LF",
    "name": "WIN-UV5OA3SO3LF",
    "version": "5.2.0"
  },
  "input_type": "log",
  "message": "Exception in thread \"main\" java.lang.IllegalStateException: A book has a null property\n       at com.example.myproject.Author.getBookIds(Author.java:38)\n       at com.example.myproject.Bootstrap.main(Bootstrap.java:14)\nCaused by: java.lang.NullPointerException\n       at com.example.myproject.Book.getId(Book.java:22)\n       at com.example.myproject.Author.getBookIds(Author.java:35)\n       ... 1 more",
  "offset": 409,
  "source": "C:\\Filebeat\\test\\testLog.txt",
  "type": "log"
}

2017-05-15T08:21:41-07:00 DBG  Publish: {
  "@timestamp": "2017-05-15T15:21:31.768Z",
  "beat": {
    "hostname": "WIN-UV5OA3SO3LF",
    "name": "WIN-UV5OA3SO3LF",
    "version": "5.2.0"
  },
  "input_type": "log",
  "message": "Exception in thread \"main\" java.lang.NullPointerException\n        at com.example.myproject.Book.getTitle(Book.java:16)\n        at com.example.myproject.Author.getBookTitles(Author.java:25)\n        at com.example.myproject.Bootstrap.main(Bootstrap.java:14)",
  "offset": 669,
  "source": "C:\\Filebeat\\test\\testLog.txt",
  "type": "log"
}

Problem:

Multiline events parsed by filebeat is not getting tagged as multiline.
Due to this my grok filter on logstash is not able to treat it accordingly



    if "multiline" in [tags] {
      grok {
        match => ["message", "%{JAVASTACKTRACEPART}"]
      }
    }

Any pointers will be appreciated!

Do you have some parts inside your filter that is executed only with multiline? You could try a "brute-force" solution, inserting `mutate { add_tag => "multiline" }` where it is needed — Andrea, May 16 '17 at 09:38
Thanks for the possible solution that I can try to unblock myself. But as a long term fix, I do prefer to understand what is missing in my approach and fix that. — Nilesh Rajani, May 16 '17 at 12:50
I definitely agree with you. This is meant to be a temporary "quick" fix, not a good long term solution. — Andrea, May 16 '17 at 14:22

score 1 · Answer 1 · answered Oct 24 '19 at 00:38

Sorry for I do not answer to question instead I would suggest looking to problem from another side: why not introduce the fixed pattern for the beginning of each new line of log (that is common practice)?

That allows using negate-mode for multiline-feature which is easier and more straight forward way.

Let's accept that each new line starts with a date, like this:

2019.10.23 01:01:01.384500 [ERROR] Exception in thread "main" java.lang.IllegalStateException: A book has a null property
       at com.example.myproject.Author.getBookIds(Author.java:38)
       at com.example.myproject.Bootstrap.main(Bootstrap.java:14)
Caused by: java.lang.NullPointerException
       at com.example.myproject.Book.getId(Book.java:22)
       at com.example.myproject.Author.getBookIds(Author.java:35)
       ... 1 more
2019.10.23 01:02:01.384500 [ERROR] Exception in thread "main" java.lang.NullPointerException
        at com.example.myproject.Book.getTitle(Book.java:16)
        at com.example.myproject.Author.getBookTitles(Author.java:25)
        at com.example.myproject.Bootstrap.main(Bootstrap.java:14)

Now, it is needed just to define the pattern for beginning each new line and turn on the negate-mode:

filebeat.inputs:
- type: log
  enabled: true
  # https://www.elastic.co/guide/en/beats/filebeat/current/multiline-examples.html
  # This regexp describes the begin of log row (for example, '2019.10.23 01:01:01.384500 ..').
  multiline.pattern: '^\d{4}.\d{2}.\d{2} \d{2}:\d{2}:\d{2}'
  multiline.negate: true
  multiline.match: after  
  paths:
    - ..

processors:
   - ..
   - dissect:
      # tokenizer syntax: https://www.elastic.co/guide/en/logstash/current/plugins-filters-dissect.html.
      tokenizer: "%{timestamp} [%{level}] %{?message}"
      # https://www.elastic.co/guide/en/beats/filebeat/master/dissect.html
      field: "message"
      target_prefix: ""

Filebeat multiline tag is not getting added for multiline event

1 Answers1