Querying Kibana using grok pattern

Question

We have configured ELK stack over our daily logs and using Kibana UI to perform basic search/query operation on the the set of logs.

Some of our logs have a certain field in the message while others don't. Therefore we have not configured it as a separate field while configuring Logstash.

I have logs like:

[28/Jun/2016:23:59:56 +0530] 192.168.xxx.xxx [API:Profile]get_data_login: Project password success:  9xxxxxxxxx0
[28/Jun/2016:23:59:56 +0530] 192.168.xxx.xxx [API:Profile]session_end: logout success:  9xxxxxxxxx0 TotalTime:1.1234

In these two logs, I wish to extract TotalTime for all session_end logs. And visualize it.

How should I do it?

I can search all the logs which are listed under session_end, however I am not able to perform grok on the set of logs.

Karup · Accepted Answer · 2016-06-29T13:43:12.360

3

Inside your filter in logstash you can have something like :

filter {

    ...

    if ([message] ~= "session_end") {
        grok {
             #write grok specifically for the second format of log here
        }
    }
    else if ([message] ~= "get_data_login") {
        grok {
             #write grok specifically for the first format of log here
        }
    }

    ...

}

Grok patterns cannot be used for querying in Kibana.

edited Jun 29 '16 at 13:43

answered Jun 29 '16 at 13:27

Karup

2,024
3
22
48

It might be more efficient to do the filtering with positive comparison, like: if ([message] ~= "session_end:" and [message] ~= "TotalTime:") – baudsp Jun 29 '16 at 13:40

baudsp · Answer 2 · 2016-06-29T13:45:31.787

You can use two different grok patterns in the same filter:

grok {
  match => {
    "message" => ['\[%{HTTPDATE}\] %{IP} \[API:Profile\]session_end: %{GREEDYDATA:session} TotalTime:%{GREEDYDATA:tt}',
                '\[%{HTTPDATE}\] %{IP} \[API:Profile\]%{GREEDYDATA:data}']
  }
}

The messages will be tested by the first pattern, if they have session_end: and TotalTime:, you'll have an elasticsearch document with the two fields. Then you'll be able to do aggregations and visualisation on these fields.

The other messages (without session_end: and TotalTime:) will be parsed by the second filter.

Querying Kibana using grok pattern

2 Answers2