Serilog HTTP sink + Logstash: Splitting Serilog message array into individual log events

Question

We're using Serilog HTTP sink to send the messages to Logstash. But the HTTP message body is like this:

{
  "events": [
    {
      "Timestamp": "2016-11-03T00:09:11.4899425+01:00",
      "Level": "Debug",
      "MessageTemplate": "Logging {@Heartbeat} from {Computer}",
      "RenderedMessage": "Logging { UserName: \"Mike\", UserDomainName: \"Home\" } from \"Workstation\"",
      "Properties": {
        "Heartbeat": {
          "UserName": "Mike",
          "UserDomainName": "Home"
        },
        "Computer": "Workstation"
      }
    },
    {
      "Timestamp": "2016-11-03T00:09:12.4905685+01:00",
      "Level": "Debug",
      "MessageTemplate": "Logging {@Heartbeat} from {Computer}",
      "RenderedMessage": "Logging { UserName: \"Mike\", UserDomainName: \"Home\" } from \"Workstation\"",
      "Properties": {
        "Heartbeat": {
          "UserName": "Mike",
          "UserDomainName": "Home"
        },
        "Computer": "Workstation"
      }
    }
  ]
}

ie. the logging events are batched in an array. It is possible to send the messages one by one, but it's still a one-item array then.

The event is then displayed in Kibana as having field message with value

{
  "events": [
    {
      // ...
    },
    {
      // ...
    }
  ]
}

ie. literally what came from the HTTP input.

How can I split the items in the events array to individual logging events and "pull up" the properties to the top level so that I would have two logging events in ElasticSearch:

  "Timestamp": "2016-11-03T00:09:11.4899425+01:00",
  "Level": "Debug",
  "MessageTemplate": "Logging {@Heartbeat} from {Computer}",
  "RenderedMessage": "Logging { UserName: \"Mike\", UserDomainName: \"Home\" } from \"Workstation\"",
  "Properties": {
    "Heartbeat": {
      "UserName": "Mike",
      "UserDomainName": "Home"
    },
    "Computer": "Workstation"
  }

  "Timestamp": "2016-11-03T00:09:12.4905685+01:00",
  "Level": "Debug",
  "MessageTemplate": "Logging {@Heartbeat} from {Computer}",
  "RenderedMessage": "Logging { UserName: \"Mike\", UserDomainName: \"Home\" } from \"Workstation\"",
  "Properties": {
    "Heartbeat": {
      "UserName": "Mike",
      "UserDomainName": "Home"
    },
    "Computer": "Workstation"
  }

I tried Logstash json and split, but I can't make it work.

score 4 · Answer 1 · answered Jan 20 '17 at 04:53

You can achieve what you expect using an additional ruby filter to pull up the fields from the sub-structure:

filter {
  split {
   field => "events"
  }
  ruby {
    code => "
       event.to_hash.update(event['events'].to_hash) 
       event.to_hash.delete_if {|k, v| k == 'events'}     
    "
  }
}

The resulting event will look like this:

{
           "@version" => "1",
         "@timestamp" => "2017-01-20T04:51:39.223Z",
               "host" => "iMac.local",
          "Timestamp" => "2016-11-03T00:09:12.4905685+01:00",
              "Level" => "Debug",
    "MessageTemplate" => "Logging {@Heartbeat} from {Computer}",
    "RenderedMessage" => "Logging { UserName: \"Mike\", UserDomainName: \"Home\" } from \"Workstation\"",
         "Properties" => {
        "Heartbeat" => {
                  "UserName" => "Mike",
            "UserDomainName" => "Home"
        },
         "Computer" => "Workstation"
    }
}

Thanks, that helped a lot. I had to start by adding `json { source => "message" }` to pull up the `events` as an object to the root of the logging event - apparently Serilog sends the message without the `application/json` content type, so the whole HTTP body is dumped as a string in the `message` field. After that your solution worked perfectly. — Adam Michalik, Jan 20 '17 at 08:29
Oh, I missed the codec solution! I will actually use the `json` codec, feels cleaner. Thanks! — Adam Michalik, Jan 20 '17 at 08:40

score 1 · Accepted Answer · answered Oct 23 '17 at 13:37

1

After upgrading to Logstash 5.0 Val's solution stopped working due to a change in the Event API: updating event.to_hash was not reflected in the original event. For Logstash 5.0+ event.get('field') and event.set('field', value) accessors must be used.

The updated solution is now:

input {
  http {
    port => 8080
    codec => json
  }
}

filter {
  split {
    field => "events"
  }
  ruby {
    code => "
      event.get('events').each do |k, v|
        event.set(k, v)
      end
    "
  }
  mutate {
    remove_field => [ "events" ]
  }
}

answered Oct 23 '17 at 13:37

Adam Michalik

9,678
13
71
102

You should have updated your question to mention the version of Logstash where the previous solution worked and created a new question with the new version and your new solution. – Val Oct 23 '17 at 14:52
@Val I don't think that's the general consensus: https://meta.stackoverflow.com/q/265433/466738 https://meta.stackoverflow.com/q/268466/466738 . I do appreciate your answer for Logstash 2.x, though! – Adam Michalik Oct 23 '17 at 15:26
Fair enough, no worries – Val Oct 23 '17 at 15:27

score 1 · Answer 3 · answered Jan 24 '20 at 07:30

You can now achieve this by setting batchFormatter. Default batch formatter will create faulty events, but ArrayBatchFormatter will fix this:

 logger.WriteTo.DurableHttpUsingFileSizeRolledBuffers(
                    requestUri: new Uri($"http://{elasticHost}:{elasticPort}").ToString(),
                    batchFormatter: new ArrayBatchFormatter());

Serilog HTTP sink + Logstash: Splitting Serilog message array into individual log events

3 Answers3

Linked