0

I am trying to get the Timestamp,AppName (mm1-spring-music.example.com), and Proc id present in log as [RTR], AppID (present in log after app_id:),Response Time (present in log after response_time:) from following Log based on Syslog 5424 Format from Logstash

2015-08-03T09:51:15.000+00:00 [RTR] OUT mm1-spring-music.example.com - [03/08/2015:09:51:15 +0000] "GET /assets/templates/status.html HTTP/1.1" 200 428 "http://mm1-spring-music.example.com/" "Mozilla/5.0 (Windows NT 6.3; WOW64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/44.0.2403.125 Safari/537.36" X.X.X.XX:37610 x_forwarded_for:"X.X.X.XX, X.X.X.XX" vcap_request_id:5ad09855-f9f3-46a9-7fa7-2095952faf78 response_time:0.002043376 app_id:08be9fc8-c7a3-4613-bf12-1a9c7d98cc27

i am very new to logstash use the following configuration i am unable to get the desired output any suggestion would of a great help 

input {
  stdin{}
}
filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOG5424PRI}%{NONNEGINT:syslog5424_ver} +(?:%{TIMESTAMP_ISO8601:syslog5424_ts}|-) +(?:%{HOSTNAME:syslog5424_host}|-) +(?:%{NOTSPACE:syslog5424_app}|-) +(?:%{NOTSPACE:syslog5424_proc}|-) +(?:%{WORD:syslog5424_msgid}|-) +(?:%{SYSLOG5424SD:syslog5424_sd}|-|) +%{GREEDYDATA:syslog5424_msg}" }
    }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
    if !("_grokparsefailure" in [tags]) {
      mutate {
        replace => [ "@source_host", "%{syslog_hostname}" ]
        replace => [ "@message", "%{syslog_message}" ]
      }
    }
    mutate {
      remove_field => [ "syslog_hostname", "syslog_message", "syslog_timestamp" ]
    }
  }
}
output {
 # elasticsearch {
  #  host => "X.X.0.103"
#   protocol => "http"
 # }

stdout { codec => rubydebug }
#stdout { }
}
Amit_Hora
  • 716
  • 1
  • 8
  • 27

2 Answers2

1

I suggest to use this website in order to test the grok filter : http://grokconstructor.appspot.com/do/match

mherbert
  • 515
  • 3
  • 12
0

Your sample input starts with a date, yet your sample grok pattern starts with a syslog priority. As you continue forward, I don't see the brackets around the hostname being dealt with.

The best idea with grok is to start at the debugger. Work from left to right, one field at a time, getting each one to work before moving on.

Alain Collins
  • 16,268
  • 2
  • 32
  • 55
  • Thanks for the suggestion After doing N number of tries i was able to get information using following configuration grok { match => { "message"=>"(?:%{TIMESTAMP_ISO8601:syslog5424_ts}|-)(?:%{NOTSPACE:syslog5424_dump}|-) (?:\[%{NOTSPACE:syslog5424_proc}\]) (?:%{NOTSPACE:syslog5424_msgtype}|-) (?:%{NOTSPACE:syslog5424_appname}|-)"} add_field => [ "received_at", "%{@timestamp}" ] add_field => [ "received_from", "%{host}" ] } – Amit_Hora Aug 04 '15 at 08:55