3

I have just put up an ELK stack, but I am having trouble regarding the logstash configuration in /etc/logstash/conf.d I have two input sources being forwarded from one linux server, which has a logstash forwarder installed on it with the "files" looking like:

{
      "paths": ["/var/log/syslog","/var/log/auth.log"],
      "fields": { "type": "syslog" }
    },
    { 
      "paths": ["/var/log/osquery/osqueryd.results.log"],
      "fields": { "type": "osquery_json" } 
}

As you can see, one input is an osquery output (json formatted), and the other is syslog. My current config for logstash is osquery.conf:

input {
  lumberjack {
    port => 5003
    ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
    ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
    codec => "json"
  }
}

filter {
   if [type] == "osquery_json" {
      date {
        match => [ "unixTime", "UNIX" ]
      }
   }
}

output {
  elasticsearch { host => localhost }
  stdout { codec => rubydebug }
}

Which works fine for the one input source, but I do not know how to add my other syslog input source to the same config, as the "codec" field is in the input -- I can't change it to syslog...

I am also planning on adding another input source in a windows log format that is not being forwarded by a logstash forwarder. Is there anyway to structure this differently?

oguz ismail
  • 1
  • 16
  • 47
  • 69
jeffrey
  • 3,196
  • 7
  • 26
  • 44

1 Answers1

2

It's probably better to just remove the codec from your input if you are going to be handling different codecs on the same input:

input {
  lumberjack {
    port => 5003
    ssl_certificate => "/etc/pki/tls/certs/logstash-forwarder.crt"
    ssl_key => "/etc/pki/tls/private/logstash-forwarder.key"
  }
}

filter {
   if [type] == "osquery_json" {
      json {
        source => "field_name_the_json_encoded_data_is_stored_in"
      }
      date {
        match => [ "unixTime", "UNIX" ]
      }
   }
   if [type] == "syslog" {

   }
}

output {
  elasticsearch { host => localhost }
  stdout { codec => rubydebug }
}

Then you just need to decide what you want to do with your syslog messages.

I would suggest also splitting your config into multiple files. I tend to to use 01-filename.conf - 10-filename.conf for inputs, 11-29 as filters and anything above that for outputs. These files will be loaded in to logstash in the order they are printed in an ls.

Rumbles
  • 1,367
  • 3
  • 16
  • 40
  • I see, if you split the config into multiple files, logstash will automatically associate same named files (it will know to associate the input, filter, and output based on filename)? This would be great as I could just create a 2 files for json, and 2 for syslog, with one unified output file for both. – jeffrey Apr 29 '15 at 17:03
  • It doesn't do that even if the config is all in one file. If you write a filter and don't define the types it should be applied against, it will apply it against all events, even that are loaded from different inputs. The only way logstash knows whether to apply a filter against an event is the conditionals – Rumbles Apr 29 '15 at 17:08
  • Hmm, then what is the point of splitting the config into multiple files? The problem on my end is that I cannot process the logs without a codec field in the input. Not to mention that I have additional logs coming from different ports. Perhaps this can be solved by having separate input files and issuing a type field in the input? – jeffrey Apr 29 '15 at 17:32
  • The point of having the config in separate files is just to make it easy to keep all the actions for each "type" separated, that way if you want to remove the config for a type you can just move the file out of the dir then restart logstash and it won't be applied any more. You don't need a codec in your input, that is handled by the filter. You can define the type in the input if you like, but if logstash-forwader is in use as in this case you can set the type for each different file you are loading in with the LSF config, not the input config. That way one input can handle multiple types. – Rumbles Apr 29 '15 at 18:00
  • In fact in your OPP you show that you are defining the type in your LSF config – Rumbles Apr 29 '15 at 18:01