Parse integer and show in timeline in Kibana

Question

Here is the Kibana UI and I want to parse some Integer in the message. The number in the end of message is the process time for one method and I what to visualize the average process time by hour in Kibana. Is that possible?

I tried some conf in logstash:

filter{
    json{
        source => "message"
    }
    grok {
        match => {
            "message" => "^Finish validate %{NUMBER:cto_validate_time}$"
        }
    }
    grok {
        match => {
            "message" => "^Finish customize %{NUMBER:cto_customize_time}$"
        }
    }

}

It works. But when I create the timechart I can not get the new field.

Answer 1

Since you don't care about performance issues, you may create a scripted field named process_time in your index pattern with the following painless code. What it does is simply take the last numerical value from your message field.

def m = /.*\s(\d+)$/.matcher(doc['message.keyword'].value);
if ( m.matches() ) {
   return m.group(1)
} else {
   return 0
}

Then you can build a chart to show the average process time by hour. Go to the Visualize tab and create a new vertical bar chart. On the Y-Axis you'll create an Average aggregation on the process_time field and on the X-Axis you'll use a Date histogram aggregation on your timestamp field. A sample is shown below:

Note: You also need to add the following line in your elasticsearch.yml file and restart ES:

script.painless.regex.enabled: true

UPDATE

If you want to do it via Logstash you can add the following grok filter

filter{
    grok {
        match => {
            "message" => "^Finish customize in controller %{NUMBER:cto_customize_time}$"
        }
    }
    mutate {
        convert => { "cto_customize_time" => "integer" }
    }
}