Split log message on space for grok pattern

Question

I am two days new to grok and ELK. I am struggling with breaking up the log messages based on space and make them appear as different fields in the logstash.

My input pattern is: 2022-02-11 11:57:49 - app - INFO - function_name=add elapsed_time=0.0296 input_params=6_3

I would like to see different fields in the logstash/kibana for function_name, elapsed_time and input_params.

At the moment, I have a following .conf

input{
  file{
  path => "/path/to/log/file"
  start_position => "beginning"
  }
}
filter{
  grok{
  match => {"message" => "%{TIMESTAMP_ISO8601:timestamp} %{LOGLEVEL:log-level} %{(?<function_name>[^.]*)\.(?<elapsed_time>[^.]*)\.(?<input>[^.]*)}"}
  }
    date {
    match => ["timestamp", "ISO8601"]
    }
    function_name {
    match => ["function_name", "DATA"]
    }
    elapsed_time {
    match => ["elapsed_time", "BASE16FLOAT"]
    }
    input {
    match => ["input", "DATA"]
    }
}
output{
  elasticsearch{
  hosts => ["localhost:9200"]
  index => "math_apis"
  }
  stdout{codec => rubydebug}
}

But this only produces a following message in logstash

{
          "host" => "hostname",
    "@timestamp" => 2022-02-11T06:27:49.404Z,
       "message" => "2022-02-11 11:57:49 - app - INFO - function_name=add elapsed_time=0.0296 input_params=6_3",
          "path" => "path/to/log/file",
      "@version" => "1",
          "tags" => [
        [0] "_grokparsefailure"
    ]
}

score 2 · Accepted Answer · answered Feb 11 '22 at 08:15

You can use the following pattern:

%{TIMESTAMP_ISO8601:timestamp} - \S+ - %{LOGLEVEL:log_level} - function_name=%{NOTSPACE:function_name} elapsed_time=%{NOTSPACE:elapsed_time} input_params=%{NOTSPACE:input}

Details:

%{TIMESTAMP_ISO8601:timestamp} - timestamp field
- - a literal string
\S+ - any one or more non-whitespace chars
- - a literal string
%{LOGLEVEL:log_level} - LOGLEVEL pattern
- function_name= - a literal string
%{NOTSPACE:function_name} - function_name field of one or more non-whitespace chars
elapsed_time= - space and elapsed_time= string
%{NOTSPACE:elapsed_time} - elapsed_time field of one or more non-whitespace chars
input_params= - literal string
%{NOTSPACE:input} - input field of one or more non-whitespace chars.

See more about Grok patterns here.

Test output:

{
  "timestamp": [
    [
      "2022-02-11 11:57:49"
    ]
  ],
  "YEAR": [
    [
      "2022"
    ]
  ],
  "MONTHNUM": [
    [
      "02"
    ]
  ],
  "MONTHDAY": [
    [
      "11"
    ]
  ],
  "HOUR": [
    [
      "11",
      null
    ]
  ],
  "MINUTE": [
    [
      "57",
      null
    ]
  ],
  "SECOND": [
    [
      "49"
    ]
  ],
  "ISO8601_TIMEZONE": [
    [
      null
    ]
  ],
  "log_level": [
    [
      "INFO"
    ]
  ],
  "function_name": [
    [
      "add"
    ]
  ],
  "elapsed_time": [
    [
      "0.0296"
    ]
  ],
  "input": [
    [
      "6_3"
    ]
  ]
}

Split log message on space for grok pattern

1 Answers1