can't parse xml input with logstash filter

Question

Hi I'm trying to parse the following xml :

<msg time='2014-08-04T14:36:02.136+03:00' org_id='oracle' comp_id='rdbms'
 msg_id='opistr_real:953:3971575317' type='NOTIFICATION' group='startup'
 level='16' host_id='linux4_l' host_addr='127.0.0.1'
 pid='8986' version='1'>
 <txt>Starting ORACLE instance (normal)
 </txt>
</msg>

using this configuration :

 input {
   stdin {
    type => "stdin-type"
  }
  }
 filter { multiline {
                       pattern => "^\s|</msg>|^[A-Za-z].*"
                        what => "previous"
                }
                xml {
                        store_xml => "false"
                        source => "message"
                        xpath => [
                                "/msg/@client_id", "msg_client_id",
                                "/msg/@host_id", "msg_host_id",
                                "/msg/@host_addr", "msg_host_addr",
                                "/msg/@level", "msg_level",
                                "/msg/@module", "msg_module",
                                "/msg/@msg_id", "msg_msg_id",
                                "/msg/@pid", "msg_pid",
                                "/msg/@org_id", "msg_org_id",
                                "/msg/@time", "msg_time",
                                "/msg/@level", "msg_level",
                                "/msg/txt/text()","msg_txt"
                        ]
               }
                date {
                        match => [ "msg_time", "ISO8601" ]
                }
                mutate {
                        add_tag => "%{type}"
                }
}
output { elasticsearch { host => localhost } stdout { codec => rubydebug } }

but when i run logstash I get the following error :

{:timestamp=>"2014-09-04T17:28:39.428000+0300", :message=>"Exception in filterworker", "exception"=>#<NoMethodError: undefined method `split' for ["msg_level", "msg_level"]:Array>, "backtrace"=>["/opt/logstash/lib/logstash/util/accessors.rb:19:in `parse'", "/opt/logstash/lib/logstash/util/accessors.rb:15:in `get'", "/opt/logstash/lib/logstash/util/accessors.rb:59:in `store_path'", "/opt/logstash/lib/logstash/util/accessors.rb:55:in `lookup'", "/opt/logstash/lib/logstash/util/accessors.rb:34:in `get'", "/opt/logstash/lib/logstash/event.rb:127:in `[]'", "/opt/logstash/lib/logstash/filters/xml.rb:117:in `filter'"

.... "/opt/logstash/lib/logstash/pipeline.rb:143:in `start_filters'"], :level=>:error} {:timestamp=>"2014-09-04T17:30:47.805000+0300", :message=>"Interrupt received. Shutting down the pipeline.", :level=>:warn}

As you pasted it in, the "xml" isn't really XML. You've got a `txt` element appearing inside of where the attributes for `msg` are. Shouldn't that xml be this: ` ` (note the > before ) — Alcanzar, Sep 04 '14 at 20:45
taking out the multiline and elasticsearch and just testing the rest works for me: `echo "Starting ORACLE instance (normal) " | bin/logstash -f test5.conf` generates an event:`{ "message" => "Starting ORACLE instance (normal) ", "@version" => "1", "@timestamp" => "2014-08-04T11:36:02.136Z", "msg_org_id" => [ [0] "oracle" ], ...` — Alcanzar, Sep 04 '14 at 20:52
@Alcanzar you are correct I've fixed the copy/paste mistake . with the XML , I guess it is problem with the multiline codec expressing . How can I debug it ? — agonen, Sep 05 '14 at 06:45
I'd take out the xml/date/mutate and just go with the rubydebug output and see what the multiline is giving you to begin with. — Alcanzar, Sep 05 '14 at 13:06

score 1 · Accepted Answer · answered Sep 07 '14 at 06:13

1

I've found my problem I've duplicated parsing on xpath , /msg@level apper twice .

answered Sep 07 '14 at 06:13

agonen

311
3
10

score 0 · Answer 2 · answered Sep 05 '14 at 14:07

0

The multiline codec is not well suited for this type of file, but you'd use something like:

multiline {
      pattern => '<msg'
      negate => true
      what => previous
}

It has the problem that the last event in the file doesn't go out until the next event comes in (so you end up losing the last event in a file).

answered Sep 05 '14 at 14:07

Alcanzar

16,985
6
42
59

thanks @Alcanzar you've put me on the right track , appently my main problem with in wrong xpath input (twice /msg@level) – agonen Sep 07 '14 at 06:14

score 0 · Answer 3 · answered Sep 12 '16 at 11:59

It has the problem that the last event in the file doesn't go out until the next event comes in (so you end up losing the last event in a file).

It's better to match the closing tag instead.

multiline {
    pattern => "</msg>$"
    negate => true
    what => next
}

can't parse xml input with logstash filter

3 Answers3

Linked