1

At first I displayed the logs in Kibana from the syslog and it worked fine. I set it up according to the documentation.

Now I've changed the source of the logs, now it retrieves logs from my web application and although Kibana still displays them kind of correctly, now there're the Tags "_grokparsefailure" which means that there's an error in parsing the logs.

The current filter I have:

filter {
  if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}

How can I find out where the parsing error is? Is there an online service which would help me create a grok pattern for my new logs? Any other advice?

UPDATE: the logs are in json.

Sojo
  • 5,455
  • 3
  • 10
  • 11
  • Are you asking for help parsing json logs in logstash? Providing an example log or two can really go a long way towards helping you. – Will Barnwell Jun 07 '16 at 15:38
  • Also you have 3 questions and an edit, which you seem to be asking for help about, please clearly ask one question so that we can address that question – Will Barnwell Jun 07 '16 at 15:39

2 Answers2

2

In response to OP's:

Is there an online service which would help me create a grok pattern for my new logs?

My favorite tool for testing grok patterns is: http://grokconstructor.appspot.com/do/match

But I know some prefer: https://grokdebug.herokuapp.com/

Your logs probably aren't parsing properly because you're using the syslog pattern on logs that aren't in the syslog format.

EDIT: For json log parsing you may want to look at either the json filter or the json codec

Will Barnwell
  • 4,049
  • 21
  • 34
0

You can debug your logs in Kibana

GoTo: Kibana -> Managenement -> Dev Tools -> Grok Debuggergrok debugger in kibana v 7.8.0

sergpank
  • 988
  • 10
  • 18