Parsing JSON file into logstash

Question

Hi I am trying to send a json file with multiple objects to elasticsearch with the logstash so I can display the data using kibana. I have researched this extensively and simply cannot understand how to make the data formatted correctly to be used in kibana.

I have tried to use different filters such as: json, date, and grok

The issue is probably how I'm going about using these filters as I can't understand it's setup all to well.

Here is a sample line of the input json file:

{"time":"2015-09-20;12:13:24","bug_code":"tr","stacktrace":"543534"},

I want to use this format for displaying the data in kibana and sorting many objects according to their "time"

this following is what my current filter section is:

filter {
    date {
        match => ["time", "YYYY-MM-dd;HH:mm:ss Z" ]
        timezone => "America/New_York"
        locale => "en"
        target => "@timestamp"
    }
    grok {
        match => ["time", "%{TIMESTAMP_ISO8601:timestamp}"]
    }
}

At this point I know the grok is wrong because I get "_grokparsefailure" but how can I figure out the correct way to use grok or is there a simple way to sort the data using the given timestamp and not the processed timestamp given when sending the data through.

here is what the output currently shows:

"message" => "{\"time\":\"2015-09-20;12:13:24\",\"bug_code\":\"tr\",\"stacktrace\":\"543534\"},\r",
"@version" => "1",
"@timestamp" => "2015-11-23T09:54:50:274Z",
"host" => "<my_computer>",
"path" => "<path_to_.json>",
"type" => "json",
"tags" => [
[0] "_grokparsefailure"

any advice would be very much appreciated

The first step would be process the json. Try the json input codec, or the json filter. Check the results with a stdout{} output. Once that's working, add in the other stuff. — Alain Collins, Nov 23 '15 at 17:55

score 2 · Answer 1 · answered Nov 24 '15 at 04:46

You're almost there, I could get it working with a few tweaks.

First, you need to add the json{} filter in the first position. Then you need to change the date pattern to YYYY-MM-dd;HH:mm:ss and finally you can remove the grok filter at the end. You filter configuration would look like this:

filter {
    json {
        source => "message"
    }
    date {
        match => ["time", "YYYY-MM-dd;HH:mm:ss" ]
        timezone => "America/New_York"
        locale => "en"
        target => "@timestamp"
    }
}

The parsed event for your sample JSON line would then look like this:

{
       "message" => "{\"time\":\"2015-09-20;12:13:24\",\"bug_code\":\"tr\",\"stacktrace\":\"543534\"}",
      "@version" => "1",
    "@timestamp" => "2015-09-20T16:13:24.000Z",
          "host" => "iMac.local",
          "time" => "2015-09-20;12:13:24",
      "bug_code" => "tr",
    "stacktrace" => "543534"
}

Parsing JSON file into logstash

1 Answers1

Linked