Decompose Logstash json message into fields

Question

It have a logfile that stores event with a timestamp and a json message. For example:

timestamp {"foo": 12, "bar": 13}

I would like to decompose the keys (foo and bar) in the json part into fields in the Logstash output.

I'm aware that I can set the format field in the Logstash file filter to json_event but in that case I have to include the timestamp in json. There is also a json filter, but that adds a single field with the complete json data structure, instead of using the keys.

Any ideas how this can be done?

You definetely should give a try to @Nikhil-S' answer. Alternatively you can also use a kv filter http://logstash.net/docs/1.4.1/filters/kv — Aldian, Jun 16 '14 at 08:27

score 16 · Answer 1 · edited Jul 09 '14 at 16:47

16

Try the latest logstash 1.2.1 and use codec value to parse json events directly.

input {
    file {
        type => "tweetfile"
        path => ["/home/nikhil/temp/feed/*.txt"]
        codec => "json"
    }
}
filter{
    json{
        source => "message"
        target => "tweet"
    }
}
output {
    stdout { }
    elasticsearch { embedded => true }
}

edited Jul 09 '14 at 16:47

Stéphane Bruckert

21,706
14
92
130

answered Oct 01 '13 at 10:16

Nikhil S

1,179
1
9
19

What happens with the timestamp field when this is done? – antak Aug 19 '16 at 14:01

mimes70 · Answer 2 · 2013-09-07T02:32:39.067

I've done this with the following config:

filter {
  grok {
    match => ["message", "\[%{WORD}:%{LOGLEVEL}\] %{TIMESTAMP_ISO8601:tstamp} :: %{GREEDYDATA:msg}"]
  }
  date {
    match => [ "tstamp", "yyyy-MM-dd HH:mm:ss" ]
  }
  json {
    source => "msg"
  }
}

By the way, this is a config for the new version 1.2.0.

In version 1.1.13 you need to include a target on the json filter and the reference for message in the grok filter is @message.

score 1 · Answer 3 · answered Aug 08 '13 at 14:34

1

You can just use plain Grok filters (regex style filters/patterns) and assign the matched value into a variable for easy organization, filtering and searching.

An example:

((?<foo_identifier>(\"foo\"))):((?<foo_variable_value>(\d+,)))

Something along those lines.

Use the GrokDebugger to help out if you get stuck on the syntax, patterns and things you think should be matching but aren't.

Hope that helps a bit.

answered Aug 08 '13 at 14:34

Adam

1,962
2
17
30

2

I considered that option, but was hoping that it was straightforward to parse a json structure and have the keys automatically translated to field names. – Maurits Rijk Aug 09 '13 at 06:45

score -4 · Answer 4 · edited Jun 16 '14 at 08:30

-4

your JSON is wrong {"foo": 12, "bar" 13}

should be:

{"foo": 12, "bar": 13}

edited Jun 16 '14 at 08:30

Fluffeh

33,228
16
67
80

answered May 14 '14 at 17:58

Jeryl Cook

989
17
40

Decompose Logstash json message into fields

4 Answers4

Linked