Extract specific time field from timestamp in logstash

Question

I have an automatically generated @timestamp with the default format. What i would like is to extract the hour/month/weekday of the timestamp putting it in another field.

For example, now my timestamp looks like that:

@timestamp: "2015-08-26T09:04:42.284Z"

Is there any way to get the following fields?

hour: 09 (or 9)? In Number format.
month: 08(or 8, or Aug, August...) Number or string format.
weekday: Mon, Tue, Wed, ...

I want it to make a kibana4 Histogram based on the hour/day of connections, with an average metric. If there's a different way to achieve that, please tell me!

I've searched all the web for that, but I couldn't find any solution. I would appreciate any help on this.

score 12 · Accepted Answer · edited May 25 '18 at 15:08

12

Finally figured it out.

You can use the notation %{} to do that. Just put:

add_field => {"[hour]" => "%{+HH}"}
add_field => {"[weekday]" => "%{+EEE}"}

Here's a reference of symbols to use.

And that's it!

edited May 25 '18 at 15:08

Giacomo1968

25,759
11
71
103

answered Aug 26 '15 at 09:40

Bertofer

770
6
18

1

Thank you very much! I tried various other solutions recommended by others but they either didn't work, or no longer supported in the latest product version or required additional gem plugins which posed other SSL certificate related errors. This solution is elegant and works. Where did you find documentation on the use of the %{} syntax? – Sanjiv Jivan Oct 12 '16 at 17:55
The %{} syntax is used in the grok filter. For more docs, see https://www.elastic.co/guide/en/logstash/current/plugins-filters-grok.html – Bertofer Oct 14 '16 at 11:22
Did you do `add_field` in a grok or as a part of a date ? Would you be able to post a more complete example of the plugin where this code is used? – FrustratedWithFormsDesigner Jan 10 '18 at 21:30
4

Further testing with this suggests that the `%{+...}` syntax *only* works with `@timestamp`, and does not work with other datetime fields. – FrustratedWithFormsDesigner Jan 15 '18 at 18:28

score 0 · Answer 2 · answered Nov 12 '20 at 08:28

%{+HH} value's is taken from @timestamp. timestamp using UTC.

if you want the hours in local time you need to use a ruby filter.

  ruby {
    code => "tstamp = event.get('logdatetime').to_i
             event.set('logdatetime', Time.at(tstamp).strftime('%Y-%m-%d %H:%M:%S'))
             event.set('logdate', Time.at(tstamp).strftime('%Y-%m-%d'))"
  }

from: https://altinity.com/blog/2017/12/18/logstash-with-clickhouse

Extract specific time field from timestamp in logstash

2 Answers2