- List item
I am encountering an issue with Logstash 8 when processing logs from a client machine, while Logstash 6 correctly processes the same logs. Here are the log samples:
Log Sample - Logstash 8 (Issues):
{"event":{"original":"<86>2023-08-03T11:50:57.876174+03:00 ansible-hcl-controler root[139012]:1212"},"received_at":"2023-08-03T08:50:58.167253528Z","@timestamp":"2023-08-03T08:50:57.876Z","process":{"name":"root","pid":139012},"received_from_ipv4":"ansible-hcl-controler","type":"syslog","message":"<86>2023-08-03T11:50:57.876174+03:00 ansible-hcl-controler root[139012]: 1212","syslog_hostname":"ansible-hcl-controler","log":{"syslog":{"severity":{"name":"notice","code":5},"facility":{"name":"user-level","code":1}}},"syslog_message":"1212","@version":"1","all_rsyslog_fields":"{\"original\"=>\"<86>2023-08-03T11:50:57.876174+03:00 ansible-hcl-controler root[139012]: 1212\"} 86 2023-08-03T11:50:57.876174+03:00 {\"name\"=>\"root\", \"pid\"=>139012} syslog <86>2023-08-03T11:50:57.876174+03:00 ansible-hcl-controler root[139012]: 1212 ansible-hcl-controler 1212 linux","source_type":"linux"}
Log Sample - Logstash 6 (Correct):
{"syslog_facility":"security/authorization","port":57044,"syslog_facility_code":10,"received_at":"2023-08-03T08:44:30.499Z","syslog_message":"12213","syslog_severity":"informational","lab_id":"7979","type":"syslog","received_from_hostname":"10-157-182-6.es-si-os-ohn-42.eecloud.nsn-net.net","message":"<86>2023-08-03T11:44:30.318073+03:00 ansible-hcl-controler root[138951]: 12213","@timestamp":"2023-08-03T08:44:30.318Z","syslog_hostname":"ansible-hcl-controler","program":"root","pid":"138951","@version":"1","tags":["siem"],"source_type":"linux","syslog_severity_code":6}
As seen above, the "received_from_hostname" and "syslog_hostname" fields are present in Logstash 6-based logs but are missing in Logstash 8-based logs. This suggests that there might be an issue with the grok settings.
Below is the common configuration used in both Logstash versions:
if [type] == "syslog" {
mutate
{
gsub => ["message", "\n", ""]
}
grok {
match => {
"message" => [ "(?m)<%{NONNEGINT:syslog_pri}>[\s]*(?:%{SYSLOGTIMESTAMP:syslog_timestamp}|%{TIMESTAMP_ISO8601:syslog_timestamp})[\s]*(%{SYSLOGHOST:syslog_hostname})?(_%{DATA})*[\s]+%{SYSLOGPROG}%{DATA}[\s\n]%{GREEDYDATA:syslog_message}" ] }
add_field => { "source_type" => "linux" }
}
ruby {
code => "event.set('received_at', event.get('@timestamp'))"
}
grok {
match => { "host" => "(?:%{IPV4:received_from_ipv4}|%{IPV6:received_from_ipv6}|%{HOSTNAME:received_from_hostname})(:%{POSINT})?" }
}
syslog_pri { }
date {
match => [ "syslog_timestamp", "ISO8601", "MMM d HH:mm:ss", "MMM dd HH:mm:ss" ]
}
mutate {
remove_field => ["syslog_timestamp", "syslog_pri", "host"]
}
}
}
I am seeking assistance in identifying and resolving the issue with the grok pattern in Logstash 8 to ensure that the "received_from_hostname" field is correctly extracted. Any help in resolving this matter would be highly appreciated. Thank you all.
I am experiencing an issue with the grok pattern in Logstash 8, where the extraction of IP from the "received_from_hostname" field is resulting in a "_grokparsefailure." I would like to resolve this problem. In my Logstash configuration, I want the following fields in Logstash 8 also:
"received_from_hostname": "10-157-182-6.es-si-os-ohn-42.eecloud.nsn-net.net"
"syslog_hostname": "ansible-hcl-controler"
I expect the grok pattern to successfully extract the IP address from the "received_from_hostname" field. However, it is not working as expected, and I need assistance in identifying and correcting the issue in the grok pattern. I'm seeking a solution that works for both Logstash 8 and earlier versions. Thank you for your help.
