0

I have a variety of shell scripts from which I run a logger line for syslog with a message in json format:

printf '{"task_id": "%s", "seconds": %f, "success": %s}' ${task_id} ${num_seconds} ${success_bool}

This gets the following output in /var/log/syslog:

Feb  1 15:12:16 my-machine logger: {"task_id": "231232xyz", "seconds": 12.453000, "success": true}

I use the regular logstash syslog input to receive this, and the individual log is received as a regular log, with the message as a string:

"_source": {
    "message": "{\"task_id\": "231232xyz", \"seconds\": 12.453000, \"success\": true}",
    "tags": [
      "_jsonparsefailure",
      "_grokparsefailure"
    ],

I could obviously use just a regular message as 

task_id: 221232xyz, seconds: 12.453000, success: true

and use grok to extract and parse values into fields (including converting the number of seconds to a float), but I feel there should be a solution between using cee or just plain messages that would work best for me. Obviously other messages from Syslog would have a non-json message. Is parsing the contents of the syslog message as JSON feasible?

Edit, per comment's request, here's the logstash input:

input {
    syslog {
        port => 5000
        host => "0.0.0.0"
        type => "syslog"
        codec => "json"
    }
}

The grok filter was my (working) attempt to match the comma separated message and started extracted the execution time from it:

filter {
    grok {
        match => ["message", "seconds: %{NUMBER:exec_time}"
    }
    mutate {
       convert => {"exec_time" => "float"}
    }
}
Loic Duros
  • 5,472
  • 10
  • 43
  • 56
  • The jsonparsefailure tag implies that you're trying to use json (codec? filter?) in your config; please share what with us. – Alain Collins Feb 01 '16 at 20:41
  • I just added my logstash input config. This is a codec, not a filter. I'm not sure how I'd use a filter for this. – Loic Duros Feb 01 '16 at 20:45
  • Obviously using the grok/mutate as I show at the end of the post works, but it seems less future proof (if someone modifies the format of the string) than if Logstash could treat the message as JSON. – Loic Duros Feb 01 '16 at 20:58

1 Answers1

2

The following filters in my logstash conf appear to convert my json message string and extract the fields properly:

filter {
    grok {
        overwrite => ["message"]
    }
    json {
     source => "message"
    }
}

The three key/value pair I had in my JSON all appear to be of the correct type within the _source of the JSON entry, and I can now work with them as fields:

{
    "_source: {
        "task_id": "231232xyz", 
        "seconds": 12.453000, 
        "success": true
    ...
}
Loic Duros
  • 5,472
  • 10
  • 43
  • 56