How to pull specific data out of a message in LogStash

Question

I am trying to take log data from a custom application that has a well defined format. I am trying to pick out certain pieces of the data using the grok filter, but I am not having any luck. Here is a sample log:

- System.Data.SqlClient.SqlException (0x80131904): Arithmetic overflow error converting IDENTITY to data type int.
Arithmetic overflow occurred.

What I would like to do is extract out the SqlException out of the string. Here is the grok that I am using:

grok{
    match => 
          {
               "message" => 
               [
                   "(?m)%{DATE:TIMESTAMP_DATE}%{SPACE}%{TIME:TIMESTAMP_TIME}%{SPACE}%{WORD:LOG_LEVEL}%{SPACE}(?<THREAD>[^\s]+)%{SPACE}(?<HOST>[^\s]+)%{SPACE}%{GREEDYDATA:MESSAGE}",

                   "(?<EXCEPTION>[.*]+)"
               ]
           }
}

I have tried several different ways, but I guess I am not completely understanding the documentation. What I would expect to happen is all of the fields that I have extracts in the first set would include the result of the second set. In other words:

TIMESTAMP_DATE,TIMESTAMP_TIME,LOG_LEVEL,THREAD,HOST,MESSAGE,EXCEPTION

I am getting the other fields perfectly, it is just additional matching that I am missing. Any help would be appreciated. Thanks

Please tag more carefully. [tag:elki] != [tag:elk-stack]. – Has QUIT--Anony-Mousse Oct 22 '15 at 06:16 — Has QUIT--Anony-Mousse, Oct 22 '15 at 06:16

score 1 · Accepted Answer · answered Oct 22 '15 at 15:28

If you specify multiple patterns grok by default only looks checks the patterns until the first match is encountered. If you want to match against both patterns regardless of whether the first one matched or not you can change the behaviour like that:

grok{
      break_on_match => false
      match => 
      {
           "message" => 
           [
               "(?m)%{DATE:TIMESTAMP_DATE}%{SPACE}%{TIME:TIMESTAMP_TIME}%{SPACE}%{WORD:LOG_LEVEL}%{SPACE}(?<THREAD>[^\s]+)%{SPACE}(?<HOST>[^\s]+)%{SPACE}%{GREEDYDATA:MESSAGE}",

               "(?<EXCEPTION>[.*]+)"
           ]
       }
}

Check out the docs under: https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html#plugins-filters-grok-break_on_match

How to pull specific data out of a message in LogStash

1 Answers1