I am having an interesting problem.
After using http://grokdebug.herokuapp.com/ which works very well. I am finding that the following message, though it works on the above site, and works for seemingly identical messages, is not working as expected.
This is the pattern:
%{MONTHDAY}-%{MONTH}-%{YEAR} %{TIME} %{PROG:program}: %{LOGLEVEL:loglevel}: %{USER:from} %{IP:ip}\#%{INT:port} \(%{GREEDYDATA:request}\): %{WORD:stage}(\s|\s\s)%{GREEDYDATA:drop_reason} to %{IPORHOST:to}/%{INT:subnet}
This is the output:
{
"message" => "03-Feb-2014 21:33:51.867 queries: info: client 123.123.123.123#57710 (some.dns.server.1.1.1.1.in-addr.arpa): drop response to 231.231.231.0/24",
"@version" => "1",
"@timestamp" => "2014-02-06T00:51:04.240Z",
"type" => "ns_query",
"host" => "ns2",
"path" => "/tmp/named-query.log.29"
}
I have seen it function as expected, it is just odd that these in particular are not working, if anyone has any ideas for what I may be doing that is incorrect please let me know.
Below you will also see the internals of the actual patterns file that it is comparing against:
# Parse the time stamp whis is an odd time stamp
DNS_TIME_STAMP %{MONTHDAY}-%{MONTH}-%{YEAR} %{TIME}
# Create the Header, basically the beginning of each line of the log
DNS_HEADER %{DNS_TIME_STAMP:log_timestamp} %{PROG:program}: %{LOGLEVEL:loglevel}: %{USER:from} %{IP:ip}\#%{INT:port}
# The different queires
DNS_QUERY_1 %{DNS_HEADER} \(%{IPORHOST:request}\): %{WORD:stage}: %{IPORHOST:request2} %{WORD:rq_where} %{WORD:r_type} %{DATA:flags} \(%{IPORHOST:serviced_by}\)
DNS_QUERY_2 %{DNS_HEADER} \(%{GREEDYDATA:request}\): %{WORD:stage}: %{GREEDYDATA:request2} %{WORD:rq_where} %{WORD:r_type} %{DATA:flags} \(%{GREEDYDATA}:serviced_by\)
# The different drop/slip/etc
DNS_DROP_1 %{DNS_HEADER} \(%{IPORHOST:request}\): %{WORD:stage} %{GREEDYDATA:drop_reason} to %{IPORHOST:to}/%{INT:subnet} for %{IPORHOST:requester} %{GREEDYDATA:dr_type} \(%{BASE16NUM:request_ID}\)
DNS_DROP_2 %{DNS_HEADER} \(%{IPORHOST:request}\): %{WORD:stage}(\s|\s\s)%{GREEDYDATA:drop_reason} to %{IPORHOST:to}/%{INT:subnet} %{GREEDYDATA:dr_type} \(%{BASE16NUM:request_ID}\)
DNS_DROP_3 %{DNS_HEADER} \(%{IPORHOST:request}\): %{WORD:stage}(\s|\s\s)%{GREEDYDATA:drop_reason} to %{IPORHOST:to}/%{INT:subnet}
DNS_DROP_4 %{DNS_HEADER} \(%{GREEDYDATA:request}\): %{WORD:stage}(\s|\s\s)%{GREEDYDATA:drop_reason} to %{IPORHOST:to}/%{INT:subnet}
DNS_DROP_5 %{DNS_HEADER} \(%{GREEDYDATA:request}\): %{WORD:stage}(\s|\s\s)%{GREEDYDATA:drop_reason} to %{IPORHOST:to}/%{INT:subnet} %{GREEDYDATA:dr_type} \(%{BASE16NUM:request_ID}\)
# Bringing them together for ease of use in Logstash
DNS_QUERY (%{DNS_QUERY_1}|%{DNS_QUERY_2})
DNS_DROP (%{DNS_DROP_1}|%{DNS_DROP_2}|%{DNS_DROP_3}|%{DNS_DROP_4}|%{DNS_DROP_5}|)
# The pattern we call in the logstash config
DNS (%{DNS_QUERY}|%{DNS_DROP})
