need custom fields of log through grok filter in logstash

Question

I have logstash, kibana and elasticsearch installed on my system, with this filter configuration:

    filter{
if [type] == "syslog" {
    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:syslog_timestamp} %{SYSLOGHOST:syslog_hostname} %{DATA:syslog_program}(?:\[%{POSINT:syslog_pid}\])?: %{GREEDYDATA:syslog_message}" }
      add_field => [ "received_at", "%{@timestamp}" ]
      add_field => [ "received_from", "%{host}" ]
    }
    mutate {
            add_field => {
                            "timestamp" => "%{TIME} %{MONTH} %{monthday}"
                         }
        }

    syslog_pri { }
    date {
      match => [ "syslog_timestamp", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
    }
  }
}

and receiving output on kibana as:

but I need some fields which are as follows: @timestamp @version _id _index _type _file Log Level Host Name Host IP Process Name Response Time

I tried adding Timestamp but its printing same string instead of dynamic result

Answer 1

You're confusing patterns with fields.

A pattern is a short-hand notation that represents a regular expression, such as %{WORD} as a shortcut for "\b\w+\b".

A field is where data - including information matched by patterns - is stored. It's possible to put a pattern into a field like this: %{WORD:my_field}

In your grok{}, you match with: %{SYSLOGTIMESTAMP:syslog_timestamp}, which puts everything that was matched into a single field called syslog_timestamp. This is the month, monthday, and time seen at the front of syslog messages.

Even though SYSLOGTIMESTAMP is itself defined as "%{MONTH} +%{MONTHDAY} %{TIME}", they don't have that ":name" syntax, so no fields are created for MONTH, MONTHDAY, and TIME.

Assuming that you really do want to make a new field in the format you describe, you'd need to either:

1. make a new pattern to replace all of SYSLOGTIMESTAMP that would make fields out of the pieces of information.
2. use the existing pattern to create the syslog_timestamp field as you're doing, and then grok{} that with a simple pattern to split it apart.

I'd recommend #2, so you'd end up with something like this:

grok {
      match => { "syslog_timestamp" => "%{MONTH:month} +%{MONTHDAY:monthday} %{TIME:time}" }
}

That should do it.

Please note that your field will be a string, so it won't be of any use in range queries, etc. You should use the date{} filter to replace @timestamp with your syslog_timestamp information.

Good luck.