0

My app writes logs as json to stdout. App runs inside docker. And docker logs stdout also in json format. So I have a json nested in json. Here is an example:

{"log":"{\"message\":\"Something failed\",\"thread_name\":\"http-nio-8080-exec-1\",\"level\":\"WARN\",\"level_value\":30000,\"test-mdc\":\"test-mdc-value\",\"appname\":\"myWebservice\"}\n","stream":"stdout","time":"2017-11-30T14:44:19.514Z"}

I want to to unnest log of my app and move all fields on top level. Thanks to this answer Parsing nested JSON string in Logstash I managed to parse all jsons(although I don't understand anything from this config), but my app log is still not on the top level. Here is what I got:

 {
      "@version" => "1",
          "host" => "4c11913cddf3",
    "@timestamp" => 2017-12-01T07:50:16.562Z,
       "message" => {
           "test-mdc" => "test-mdc-value",
            "appname" => "myWebservice",
              "level" => "WARN",
        "thread_name" => "http-nio-8080-exec-1",
        "level_value" => 30000,
            "message" => "Something failed"
    }
}

My config:

 input { stdin { } }

filter {
 json {
    source => "message"
    target => "message"
  }
 json {
   source => "[message][log]"
   target => "[message]"
 }
}

output { stdout { codec => rubydebug } }

Can I move all nested fields on top level overriding duplicates if there is one? Better without ruby, because I don't know it.

Artem Malinko
  • 1,761
  • 1
  • 22
  • 39

1 Answers1

0

Answering my own question. You need to remove target => "[message]". Although I still don't fully understand what is happening here, config should look like this:

input { stdin { } }

filter {
 json {
    source => "message"
    target => "message"
  }
 json {
   source => "[message][log]"
 }
}

output { stdout { codec => rubydebug } }
Artem Malinko
  • 1,761
  • 1
  • 22
  • 39