Logstash does not parse json

Question

When i see results in Kibana, i see that there are no fields from JSON, more over, message field contains only "status" : "FAILED".

Is it possible to parse fields from json and to show them in Kibana? I have following config:

input {
  file {
    type => "json"
    path => "/home/logstash/test.json"
    codec => json
    sincedb_path => "/home/logstash/sincedb"
  }
} 

output {
  stdout {}
  elasticsearch {
    protocol => "http"
    codec => "json"
    host => "elasticsearch.dev"
    port => "9200"
  }
}

And following JSON file:

[{"uid":"441d1d1dd296fe60","name":"test_buylinks","title":"Testbuylinks","time":{"start":1419621623182,"stop":1419621640491,"duration":17309},"severity":"NORMAL","status":"FAILED"},{"uid":"a88c89b377aca0c9","name":"test_buylinks","title":"Testbuylinks","time":{"start":1419621623182,"stop":1419621640634,"duration":17452},"severity":"NORMAL","status":"FAILED"},{"uid":"32c3f8b52386c85c","name":"test_buylinks","title":"Testbuylinks","time":{"start":1419621623185,"stop":1419621640826,"duration":17641},"severity":"NORMAL","status":"FAILED"}]

stringy05 · Accepted Answer · 2015-02-26T23:36:30.277

Yes. you need to add a filter to your config, something like this.

filter{
    json{
        source => "message"
    }
}

It's described pretty well in the docs here

EDIT The json codec doesn't seem to like having an array passed in. A single element works with this config:

Input:

{"uid":"441d1d1dd296fe60","name":"test_buylinks","title":"Testbuylinks","time":{"start":1419621623182,       "stop":1419621640491,"duration":17309      },      "severity":"NORMAL",      "status":"FAILED"   }

Logstash Result:

{
      "message" => "{\"uid\":\"441d1d1dd296fe60\",\"name\":\"test_buylinks\",\"title\":\"Testbuylinks\",\"time\":{\"start\":1419621623182,       \"stop\":1419621640491,\"duration\":17309      },      \"severity\":\"NORMAL\",      \"status\":\"FAILED\"   }",
     "@version" => "1",
   "@timestamp" => "2015-02-26T23:25:12.011Z",
         "host" => "emmet.local",
          "uid" => "441d1d1dd296fe60",
         "name" => "test_buylinks",
        "title" => "Testbuylinks",
         "time" => {
          "start" => 1419621623182,
           "stop" => 1419621640491,
       "duration" => 17309
   },
     "severity" => "NORMAL",
       "status" => "FAILED"

}

Now with an array:

Input

[{"uid":"441d1d1dd296fe60","name":"test_buylinks","title":"Testbuylinks","time":{"start":1419621623182,       "stop":1419621640491,"duration":17309      },      "severity":"NORMAL",      "status":"FAILED"   }, {"uid":"441d1d1dd296fe60","name":"test_buylinks","title":"Testbuylinks","time":{"start":1419621623182,       "stop":1419621640491,"duration":17309      },      "severity":"NORMAL",      "status":"FAILED"   }]

Result:

Trouble parsing json {:source=>"message", :raw=>"[{\"uid\":\"441d1d1dd296fe60\",\"name\":\"test_buylinks\",\"title\":\"Testbuylinks\",\"time\":{\"start\":1419621623182,       \"stop\":1419621640491,\"duration\":17309      },      \"severity\":\"NORMAL\",      \"status\":\"FAILED\"   }, {\"uid\":\"441d1d1dd296fe60\",\"name\":\"test_buylinks\",\"title\":\"Testbuylinks\",\"time\":{\"start\":1419621623182,       \"stop\":1419621640491,\"duration\":17309      },      \"severity\":\"NORMAL\",      \"status\":\"FAILED\"   }]", :exception=>#<TypeError: can't convert Array into Hash>, :level=>:warn}
{
      "message" => "[{\"uid\":\"441d1d1dd296fe60\",\"name\":\"test_buylinks\",\"title\":\"Testbuylinks\",\"time\":{\"start\":1419621623182,       \"stop\":1419621640491,\"duration\":17309      },      \"severity\":\"NORMAL\",      \"status\":\"FAILED\"   }, {\"uid\":\"441d1d1dd296fe60\",\"name\":\"test_buylinks\",\"title\":\"Testbuylinks\",\"time\":{\"start\":1419621623182,       \"stop\":1419621640491,\"duration\":17309      },      \"severity\":\"NORMAL\",      \"status\":\"FAILED\"   }]",
     "@version" => "1",
   "@timestamp" => "2015-02-26T23:28:21.195Z",
         "host" => "emmet.local",
         "tags" => [
       [0] "_jsonparsefailure"
   ]
}

This looks like a bug in the codec, can you change your messages to an object rather than an array?

I've read this docs, but filter did not help. The result is the same. As i understand, this definition says: message contains valid JSON object you can use as hash. It's nice. But anyway, i still do not see valid JSON in message. Only parts. — avasin, Feb 26 '15 at 23:02
I think the issue might be that your message is an array and the json codec is expecting an object at the root — stringy05, Feb 26 '15 at 23:27
wow, perhaps we can report this a bug... Anyway, you were right - filter is a solution. https://github.com/elasticsearch/logstash/issues/2702 — avasin, Feb 26 '15 at 23:42
Seems they are aware of it, from the source code: # TODO(sissel): Note, this will not successfully handle json lists # like your text is '[ 1,2,3 ]' JSON.parse gives you an array (correctly) # which won't merge into a hash. If someone needs this, we can fix it # later. — stringy05, Feb 26 '15 at 23:42
Thanks a lot for your help. Now things got clear. Anyway, i've reported in my ticket on github — avasin, Feb 26 '15 at 23:45

score 2 · Answer 2 · answered Jul 01 '15 at 18:22

Try the json_lines codec instead of json. This must have been added recently. In your particular case, you'd first want to change your output from a list of json to newline-delimited json.

http://logstash.net/docs/1.4.0/codecs/json_lines

This codec will decode streamed JSON that is newline delimited. For decoding JSON payload in the redis input for example, use the json codec instead. Encoding will emit a single JSON string ending in a '\n'

score 2 · Answer 3 · answered Apr 06 '16 at 19:22

Reading in a file containing a JSON array is way harder than it should be. Below is a working pipeline configuration

input {
  exec {
    command => "cat /path/file_containing_json_array.txt"
    codec => "json"
    interval => 3600
  }
}

output {
  stdout {
    codec => rubydebug
  }
}

Logstash does not parse json

3 Answers3

Linked