4

I've configured logstash (v1.5.0), with a simple syslog input, as follows:

input {
  syslog {
    type => syslog
    port => 5514
  }
}

filter {
  kv {}
}

output {
  elasticsearch {
    cluster => "logs"
    host => "0.0.0.0"
    protocol => "transport"
  }
}

However it seems to be failing on some of the cron logs. The following line fails to parse with a _grokparsefailure_sysloginput:

<77>Jul 22 22:01:01 ip-172-31-2-48 run-parts(/etc/cron.hourly)[2599 finished 0yum-hourly.cron

The final JSON output is:

{
  "_index": "logstash-2015.07.22",
  "_type": "syslog",
  "_id": "AU63yLrC118PBgBqQxRA",
  "_score": null,
  "_source": {
    "message": "<77>Jul 22 22:01:01 ip-172-31-2-48 run-parts(/etc/cron.hourly)[2599 finished 0yum-hourly.cron\n",
    "@version": "1",
    "@timestamp": "2015-07-22T22:01:01.569Z",
    "type": "syslog",
    "host": "172.31.2.48",
    "tags": [
      "_grokparsefailure_sysloginput"
    ],
    "priority": 0,
    "severity": 0,
    "facility": 0,
    "facility_label": "kernel",
    "severity_label": "Emergency"
  },
  "fields": {
    "@timestamp": [
      1437602461569
    ]
  },
  "sort": [
    1437602461569
  ]
}

Any pointers?

Olly
  • 449
  • 1
  • 4
  • 11
  • I don't think that's a standard Syslog line format. What's the Grok pattern you're using? – GregL Jul 23 '15 at 15:12

3 Answers3

6

The syslog input use grok internally, your message is probably not following the syslog standard 100%.

The solution in this link worked for me: http://kartar.net/2014/09/when-logstash-and-syslog-go-wrong/

The key info from the link is:

Replace the existing syslog block in the Logstash configuration with:

input {
  tcp {
    port => 514
    type => syslog
  }
  udp {
    port => 514
    type => syslog
  }
}

Next, replace the parsing element of our syslog input plugin using a grok filter plugin.

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "<%{POSINT:syslog_pri}>%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
    }
  }
}

You can edit the filter matching ("grok") syntax now, to match your desired format. It's also possible to support multiple different syntaxes with creative use of if, else if, and else.

willbradley
  • 328
  • 2
  • 6
Almaza
  • 76
  • 1
  • 1
    Please don't just post the link in an answer, but rather copy the pertinent bits and include them in the answer. Links die and pages get deleted, so keeping it all here makes the answer useful should that happen. – GregL Jul 24 '15 at 11:45
  • 1
    @GregL: edited to add the config bits from the link. – willbradley Dec 11 '15 at 01:36
3

Coming here after 4 years, now the logstash syslog input supports setting the grok pattern to use, as detailed in the documentation.

In order to keep the syslog input functionalities, one can as such insert the nonstandard pattern to parse in the grok_pattern setting, e.g.:

input {
  syslog {
    port => 514
    type => "syslog"
    grok_pattern => "(?:<%{POSINT:priority}>%{SYSLOGLINE}|YOUR NONSTANDARD PATTERN HERE)"
   }
}

or likewise amend the default <%{POSINT:priority}>%{SYSLOGLINE} pattern to make it match also the nonstandard input lines.

vjt
  • 131
  • 3
  • 2
    The docs have a great example here too https://www.elastic.co/guide/en/logstash/current/config-examples.html#_processing_syslog_messages – Everett Toews May 27 '20 at 22:27
0

I've had the same problem on logstash 7.17. Solved it by adding ecs_compatibility => "v8" in syslog input plugin configuration:

input {
    syslog {
        port => "514"
        ecs_compatibility => "v8"
    }
}
...
Viceman
  • 41
  • 2