Json transformation in Logstash

Question

I want to transform a json document from one structure to another. I've already tried to solve it with multiple workarounds but somehow I don't get my expected output.

I have a json log file as following:

 {
    "consumer": {
        "created_at": 1566912618154,
        "username": "dummyapp",
        "id": "07912445-ca35-464b-8596-b2ace5e60481"
    },
    "service": {
        "created_at": 1567173649,
        "connect_timeout": 60000,
        "protocol": "https",
        "read_timeout": 60000,
        "port": 9090,
        "updated_at": 1567173649,
        "retries": 1,
        "write_timeout": 60000
    },
    "request": {
        "querystring": {},
        "size": "361",
        "headers": {
            "cache-control": "no-cache",
            "content-length": "0",
            "postman-token": "fb9e10e4-2f66-4126-beec-d5c1f7c06bf7",
            "user-agent": "PostmanRuntime/7.15.0",
            "accept": "/",
            "accept-encoding": "gzip, deflate",
            "connection": "keep-alive"
        },
        "method": "POST"
    }
}

I want to transform it to something like this:

{
    "Title":"Sample"
    "consumer": {
        "created_at": 1566912618154,
        "username": "dummyapp"
    },
    "service": {
        "created_at": 1567173649,
        "connect_timeout": 60000
    },
    "request": {
        "querystring": {},
        "headers": {
            "user-agent": "PostmanRuntime\/7.15.0"
        }
        "method": "POST"
    }
}

This is my pipeline configuration:

input {
    file {
        path => "Path_to_Log\test.log"
        start_position => "beginning"
        type => "json"
        codec => "json"
    }
}

filter {
    mutate {
        add_field => {
            "Title" => "Sample"
        }
        add_field => {
            "consumer" => "%{[consumer]}"
        }
        ...further changes
    }
}

output {
    file {
        path => "Output_Path\Output.log"
    }
}

This is the response I get:

    { 
   "@timestamp":"2019-09-10T09:20:38.569Z",
   "Title":"Sample",
   "@version":"1",
   "consumer":[ 
      [ 
         "created_at",
         1566912618154
      ],
      [ 
         "username",
         "dummyapp"
      ],
      "{\"created_at\":1566912618154,\"username\":\"dummyappapp\"}"
   ],
   "type":"json"
   ..some data
}

• Why am I getting an output like above?
• How can I tranform json documents to fulfill the requirements I described above?
• How can I set a tag from the source to the target child element?

Answer 1

So why don't you just remove the fields that you are not interested in?

filter {
    mutate {
        add_field => {
            "Title" => "Sample"
        }

        remove_field => [
          "[consumer][id]",
          "[service][protocol]",
          # and so on for the service element
          "[request][size]",
          "[request][headers][cache-control]",
          # and so on for the request.headers element
        ]

        # THIS IS OBSOLETE NOW!
        #add_field => {
        #    "consumer" => "%{[consumer]}"
        #}
        ...further changes
    }
}

Please refer to this guide for further information on how to access event data and fields (especially nested ones).