18

I am logging to logstash,in json format, my logs have the following fields, each field is a string and the atts field is a stringified json (note: atts sub fields are different each time)

here is an example:

{"name":"bob","last":"builder", "atts":"{\"a\":111, \"b\":222}"}

I would like to parse it to something like this:

   {
     "name" => "bob",
     "last" => "builder"
     "atss" => {
          "a" => 111,
          "b" => 222}
   }

here is my configuration: 

input { stdin { } }  

filter {
  json {
    source => "message"
    target => "parsed"
  }
}
 output { stdout { codec => rubydebug  }}

ok, so now I get this:

{
    "@timestamp" => 2017-04-05T12:19:04.090Z,
    "parsed" => {
        "atss" => "{\"a\":111, \"b\":222}",
        "name" => "bob",
        "last" => "the builder"
    },
        "@version" => "1",
          "host" => "0.0.0.0"
}

how can I parse the atts field to json so I receive:

{
    "@timestamp" => 2017-04-05T12:19:04.090Z,
    "parsed" => {
        "atss" => 
           {"a" => 111,
            "b" => 222},
        "name" => "bob",
        "last" => "the builder"
    },
        "@version" => "1",
          "host" => "0.0.0.0"
}
dina
  • 4,039
  • 6
  • 39
  • 67

3 Answers3

25

thanks to @Alcanzar here is what I did

input { 
  stdin { } 
}  

filter {
  json {
    source => "message"
    target => "message"
  }
  json {
    source => "[message][atts]"
    target => "[message][atts]"
  }

}
 output { stdout { codec => rubydebug  }}
dina
  • 4,039
  • 6
  • 39
  • 67
6

There is a json filter. Just pass it the field you want to parse and a target where you want it.

Something like:

json {
  source => "[parsed][atss]"
  target => "[parsed][newfield]"
}

I'm not sure if you can put atss as new field. It might or might not work. If it doesn't, use the mutate filter to remove_field and rename_field.

Alcanzar
  • 16,985
  • 6
  • 42
  • 59
  • thanks! can you see my posted answer, is there a way to do this writing the filter all under `json` filter once? something like `filter { json { source => "message" target => "message" source => "[message][atts]" target => "[message][atts]" } }` – dina Apr 05 '17 at 14:43
1

As of August 5 2020, In logstash 7.8.0, the JSON filter now parses nested strings.
I had the following string stored in a Postgres column, lets call it "column-1"

{ "XYZ" : {"A" : "B", "C" : "D", "E" : "F"} }

My json filter looked like this

json {
    source => "column-1"
    target => "column-1"
}

And it was parsed as a nested dictionary in my Elasticsearch Index

The difference from the accepted answer is that now we just need a single json filter instead of multiple ones.

ranvit
  • 99
  • 6