I don't know how to filter my log file with grok and logstash

Question

I have an small java app that loads logs similar to these ones bellow:

Fri May 29 12:10:34 BST 2015 Trade ID: 2 status is :received
Fri May 29 14:12:36 BST 2015 Trade ID: 4 status is :received
Fri May 29 17:15:39 BST 2015 Trade ID: 3 status is :received
Fri May 29 21:19:43 BST 2015 Trade ID: 3 status is :Parsed
Sat May 30 02:24:48 BST 2015 Trade ID: 8 status is :received
Sat May 30 08:30:54 BST 2015 Trade ID: 3 status is :Data not found
Sat May 30 15:38:01 BST 2015 Trade ID: 3 status is :Book not found
Sat May 30 23:46:09 BST 2015 Trade ID: 6 status is :received

I want to use ELK stack to analyse my logs and filter them. I would like at least 3 filters : Date and time, trade Id and status.

In the filter part of my logstash configuration file here is what I did:

filter {
grok {
    match => { "message" => "%{DAY} %{MONTH} %{DAY} %{TIME} BST %{YEAR} Trade ID: %{NUMBER:tradeId}  status is : %{WORD:status}" }
  }

And for the moment I can't filter my logs as I want.

Please be more specific in your current and expected filter behavior. Saying "it doesn't work" is not helpful. — ryanyuyu, May 29 '15 at 15:40
For one, your pattern says "status is : " but there's no space after the colon in your sample data. Use the grok debugger. — Alain Collins, May 29 '15 at 16:24
For the moment my filter can only see the timestamp. there are no tags concerning tradeId and status. And I want tags on those too. I am really beginner in ELK can you please provide a working config file? — BYoussef, May 30 '15 at 17:54

score 0 · Answer 1 · answered Aug 18 '15 at 18:34

You have some extra spaces between the pattern, and for the status, you would like to parse the entire message, so using the GREEEDYDATA instead of the WORD is your choice.

filter {
    grok {
        match => { "message" => "%{DAY:day} %{MONTH:month} %{MONTHDAY:monthday} %{TIME:time} BST %{YEAR:year} Trade ID: %{NUMBER:tradeId} status is :%{GREEDYDATA:status}" }
    }
}

For this log line:

Sat May 30 15:38:01 BST 2015 Trade ID: 3 status is :Book not found

You will end up with a json like:

{
   "message" => "Sat May 30 15:38:01 BST 2015 Trade ID: 3 status is :Book not found",
  "@version" => "1",
"@timestamp" => "2015-08-18T18:28:47.195Z",
      "host" => "Gabriels-MacBook-Pro.local",
       "day" => "Sat",
     "month" => "May",
  "monthday" => "30",
      "time" => "15:38:01",
      "year" => "2015",
   "tradeId" => "3",
    "status" => "Book not found"

}

I don't know how to filter my log file with grok and logstash

1 Answers1