Logstash filtering using grok

Question

I'm trying to structurally filter my log using a grok filter in logstash.

This is a sample log:

5d563f04-b5d8-4b8d-b3ac-df26028c3719 SoapRequest CheckUserPassword <?xml version=\"1.0\" encoding=\"utf-8\"?><soap:Envelope xmlns:soap=\"http://schemas.xmlsoap.org/soap/envelope/\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xmlns:xsd=\"http://www.w3.org/2001/XMLSchema\"><soap:Body><CheckUserPassword xmlns=\"http://users.tvinci.com/\"><sWSUserName>users_199</sWSUserName><sWSPassword>11111</sWSPassword><sUserName>test</sUserName><sPassword>123456</sPassword><bPreventDoubleLogins>false</bPreventDoubleLogins></CheckUserPassword></soap:Body></soap:Envelope>

And this is my filter grok match pattern:

%{DATA:method_id} %{WORD:method_type} %{WORD:method} %{GREEDYDATA:data}

The structure I am receiving is:

  "method_id" => "963ad634-92d6-4a6c-9e6b-ef57e6bcd374",
      "method_type" => "SoapRequest",
      "method" => "CheckUserPassword",
      "data" => " <?xml version=\"1.0\" encoding=\"utf-8\"?><soap:Envelope xmlns:soap=\"http://schemas.xmlsoap.org/soap/envelope/\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xmlns:xsd=\"http://www.w3.org/2001/XMLSchema\"><soap:Body><CheckUserPassword"

WHich is the correct structure except the data field, here I expect to see the whole SOAP XML (as you can see it's cut in the middle)

Any suggestions?

try %{DATA:method_id} %{WORD:method_type} %{WORD:method} (?.*$) — Kenneth Xu, Oct 26 '13 at 03:49

score 2 · Answer 1 · edited Mar 24 '16 at 20:17

2

Use the following filter:

mutate {
  gsub => [

    "message", "\n", "",  # Unix newline

    "message", "\r", "",  # OS X newline

    "message", "\r\n", "" # Windows newline

  ]
}

edited Mar 24 '16 at 20:17

Jon

1,055
13
24

answered Sep 30 '14 at 01:19

Swapnil

31
2

This helped me out, I had a bunch of new lines messing up my XML. – Fawix Jun 30 '16 at 16:20

score 0 · Answer 2 · answered Oct 26 '13 at 04:09

Actually, you pattern should just work. It must be something else wrong in your config. Try below:

logstash.config

input {
  stdin {
  }
}

filter {
  grok {
    match => [ "message", "%{DATA:method_id} %{WORD:method_type} %{WORD:method} %{GREEDYDATA:data}" ]
  } 
}

output {
  stdout { debug => true }
}

$ java -jar logstash-1.2.1-flatjar.jar agent -f logstash.conf
5d563f04-b5d8-4b8d-b3ac-df26028c3719 SoapRequest CheckUserPassword <?xml version="1.0" encoding="utf-8"?><soap:Envelope xmlns:soap="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema"><soap:Body><CheckUserPassword xmlns="http://users.tvinci.com/"><sWSUserName>users_199</sWSUserName><sWSPassword>11111</sWSPassword><sUserName>test</sUserName><sPassword>123456</sPassword><bPreventDoubleLogins>false</bPreventDoubleLogins></CheckUserPassword></soap:Body></soap:Envelope>
{
        "message" => "5d563f04-b5d8-4b8d-b3ac-df26028c3719 SoapRequest CheckUserPassword <?xml version=\"1.0\" encoding=\"utf-8\"?><soap:Envelope xmlns:soap=\"http://schemas.xmlsoap.org/soap/envelope/\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xmlns:xsd=\"http://www.w3.org/2001/XMLSchema\"><soap:Body><CheckUserPassword xmlns=\"http://users.tvinci.com/\"><sWSUserName>users_199</sWSUserName><sWSPassword>11111</sWSPassword><sUserName>test</sUserName><sPassword>123456</sPassword><bPreventDoubleLogins>false</bPreventDoubleLogins></CheckUserPassword></soap:Body></soap:Envelope>\r",
     "@timestamp" => "2013-10-26T04:01:19.386Z",
       "@version" => "1",
           "host" => "NYCL530",
      "method_id" => "5d563f04-b5d8-4b8d-b3ac-df26028c3719",
    "method_type" => "SoapRequest",
         "method" => "CheckUserPassword",
           "data" => "<?xml version=\"1.0\" encoding=\"utf-8\"?><soap:Envelope xmlns:soap=\"http://schemas.xmlsoap.org/soap/envelope/\" xmlns:xsi=\"http://www.w3.org/2001/XMLSchema-instance\" xmlns:xsd=\"http://www.w3.org/2001/XMLSchema\"><soap:Body><CheckUserPassword xmlns=\"http://users.tvinci.com/\"><sWSUserName>users_199</sWSUserName><sWSPassword>11111</sWSPassword><sUserName>test</sUserName><sPassword>123456</sPassword><bPreventDoubleLogins>false</bPreventDoubleLogins></CheckUserPassword></soap:Body></soap:Envelope>\r"
}

score 0 · Answer 3 · answered Mar 07 '17 at 18:07

If you're having issues with new lines as suggested by another poster

From a post that I saw on the internet, it looks like this problem is also solved using the multiline codec option on the input file field.

In the below example, every time I see a TIMESTAMP_ISO8601 at the very beginning of a line, I declare that to be a new "entry," and everything up until the next timestamp at the beginning of a line is a part of that entry.

input {
  file {
    path => "/var/elasticsearch-input/Log.log"
    type => "log4netLog"
    codec => multiline {
      pattern => "^%{TIMESTAMP_ISO8601} "
      negate => true
      what => previous
    }
  }
}

In your case, you'll need to write the regex for a GUID and put it in there as the pattern. it would probably look like this, but I'm not 100% certain since I haven't tested it.

input {
  file {
    path => "/var/elasticsearch-input/Log.log"
    type => "log4netLog"
    codec => multiline {
      pattern => "^%{UUID} "
      negate => true
      what => previous
    }
  }
}

Documentation: https://www.elastic.co/guide/en/logstash/current/plugins-codecs-multiline.html Article that I pulled it from (under Configuring logstash - intputs): http://www.ben-morris.com/using-logstash-elasticsearch-and-log4net-for-centralized-logging-in-windows/

Logstash filtering using grok

3 Answers3