4

I have two somewhat similar logstash configs. One works ok and the other not and I don't know how to proceed.

First working config:

input {
  stdin {
    type => "dec"
  }
}

filter {
  if [type] == "dec" {
    if "SYN/ACK" in [message] or "ACK" in [message] {
      grok {
        match => { "message" => "%{SYSLOGTIMESTAMP:sys_timestamp} %{SYSLOGHOST:sys_hostname} %{DATA:program}: %{NUMBER:ts}:%{GREEDYDATA:reason}:%{IPV4:src_ip:ip}:%{IPV4:dst_ip:ip}:%{POSINT:src_port:int}:%{POSINT:dst_port:int}" }
      }
    } else {
      grok {
        match => { "message" => "%{SYSLOGTIMESTAMP:sys_timestamp} %{SYSLOGHOST:sys_hostname} %{DATA:program}: %{NUMBER:ts}:%{GREEDYDATA:reason}:%{IPV4:src_ip:ip}:%{IPV4:dst_ip:ip}" }
      }
    }
    date {
      timezone => "UTC"
      match => [ "ts", "UNIX_MS" ]
      target => "@timestamp"
    }

    mutate {
      remove_field => [ "message", "sys_timestamp", "sys_hostname" ]
    }

  }
}

output {
  if [type] == "dec" {

    stdout {
      codec => rubydebug
    }
  }
}

The second with the parse errors:

input {
    stdin {
    type => "lmc"
}}


filter {
  if [type] == "lmc" {

# Jun 20 17:08:11 rolf-PE-860 sds_lmc: 1529539691952:1:17:::"fe80::14f0:159d:a58d:2802":"ff02::fb":5353:5353:3
# Jun 26 10:54:58 rolf-PE-860 sds_lmc: 1530035699347:0:17:192.168.10.165:239.255.255.250:::56430:1900:3

    grok {
      match => { "message" => "%{SYSLOGTIMESTAMP:sys_timestamp} %{SYSLOGHOST:sys_hostname} %{DATA:program}: %{NUMBER:ts}:%{NUMBER:dir}:%{NUMBER:proto}:%{IPV4:src_ip:ip}:%{IPV4:dst_ip:ip}:%{IPV6:src_ip6:ip}:%{IPV6:dst_ip6:ip}:%{POSINT:src_port:int}:%{POSINT:dst_port:int}:%{DATA:act}" }
    }

    date {
      timezone => "UTC"
      match => [ "ts", "UNIX_MS" ]
      target => "@timestamp"
    }


  translate {
    field => "dir"
    destination => "direction"
    dictionary => {
      "0" => "Out"
      "1" => "In"
    }
  }

  translate {
    field => "act"
    destination => "action"
    dictionary => {
      "1" => "Allow"
      "2" => "Block"
      "3" => "Drop"
    }
  }

    mutate {
      remove_field => [ "message", "sys_timestamp", "sys_hostname", "act", "dir" ]
    }
  }
}



output {
  if [type] == "lmc" {
    stdout {
      codec => rubydebug
    }
  }
}

I've been checking logstash syslog and tried to get more info redirecting when ""_grokparsefailure" in [tags]". but I had no succsess. I laso tries to run logstash redirecting in and output, but can't get enough info.

Example on input for working config:

Jun 26 10:54:57 rolf-PE-860 sds_lmc:1530035697951:0:6:192.168.10.165:74.125.195.125:::6554:5222:3
Jun 26 10:54:58 rolf-PE-860 sds_lmc:1530035699347:0:17:192.168.10.165:239.255.255.250:::56430:1900:3

and for non working:

Jun 21 13:26:21 rolf-PE-860 ips: 1529612781461:ACK/TCP Established:192.168.10.9:192.168.10.165:23:5907
Jun 21 13:26:29 rolf-PE-860 ips: 1529612789554:ARP Response:192.168.10.127:192.168.10.140

Here is the result from the failing config. 

{
          "tags" => [
        [0] "_grokparsefailure"
    ],
          "host" => "rolf-PE-860",
      "@version" => "1",
          "type" => "lmc",
    "@timestamp" => 2018-06-26T23:41:49.349Z
}
{
          "tags" => [
        [0] "_grokparsefailure"
    ],
          "host" => "rolf-PE-860",
      "@version" => "1",
          "type" => "lmc",
    "@timestamp" => 2018-06-26T23:41:49.355Z
}

If anyone could help me out with tips on how to proceed, I'd be grateful.

Thanks

RR1
  • 333
  • 1
  • 6
  • 13
  • To debug grok filter, here's useful ressources: a [tester](http://grokconstructor.appspot.com/do/match#result) and the [default grok patterns](https://github.com/logstash-plugins/logstash-patterns-core/blob/master/patterns/grok-patterns) – baudsp Jun 27 '18 at 08:49
  • Thanks. I tried the Grokconstructor first, but I'll give it a second go since my frustration level is lower now. – RR1 Jun 27 '18 at 23:48
  • @baudsp - thanks. I used some more time and learned more about how to use the tool. This with a little patience resulted in resolving a series of issues starting with a misplaced ' '. Next time I'm better equipped to resolve more easily. – RR1 Jun 29 '18 at 00:03
  • Unfortunately, the many online grok debuggers out there won't help if your pattern works fine in the debugge, but the filter executed on logstash fails with no info besides a `_grokparsefailure` tag. – Paulo Merson Nov 25 '21 at 19:03

0 Answers0