How to parse a xml-file with logstash filters

Question

I'm trying to index some simple XML-files with elasticsearch and logstash. So far I have the ELK-stack set up, and logstash-forwarder. I am trying to use the documentation to set up a xml filter, but I just cant seem to get it right.

My XML format is pretty straigth forward;

<Recording>

  <DataFile description="desc" fileName="test.wav" Source="mic" startTime="2014-12-12_121212" stopTime="2014-12-12_131313"/>

</Recording>

I just want each file to be an entry in elasticsearch, and every parameter in the DataFile-tag to be a key-value that I can search. Since the documentation is getting me nowhere, how would such a filter look? I have also tried to use the answers in this and this without any luck.

Can you show the current Logstash config file you have so far? — Val, Jun 08 '15 at 04:15
Logstash reads files line by line, so you're first order of business would be to use the multiline filter to join the physical lines of a file into a single logical line (message). But Logstash isn't meant to read whole files like that so I'd actually look into another option. — Magnus Bäck, Jun 08 '15 at 06:14

score 1 · Accepted Answer · answered Jul 10 '15 at 10:33

Add the below in your logstash-forwarder configuration and change the logstash server IP, Certificate path and the log path accordingly.

{
"network": {
"servers": [ "x.x.x.x:5043" ],
"ssl ca": " / cert/server.crt",
"timeout": 15
},
"files": [
{
"paths": [
"D:/ELK/*.log"
],
"fields": { "type": "log" }
}
]
}

Add the below input plugin in your logstash server configuration. Change the certificate ,key path and name accordingly.

lumberjack {
  port => 5043
  type => "lumberjack"
  ssl_certificate => " /cert/server.crt"
  ssl_key => "D:/ELK/logstash/cert/server.key"
  codec => multiline {
             pattern => "(\/Recording>)"
             what => "previous"
            negate => true
            }
}

Now add the below grok filter under your logstash filter section

grok {
        match   =>  ["message", "(?<content>(< Recording(.)*?</Recording>))"]
        tag_on_failure => [ ]

        }

Finally in the logstash output session add

elasticsearch {
                    host => "127.0.0.1"
                    port => "9200"
                    protocol => "http"
                    index => "Recording-%{+YYYY.MM.dd}"
                    index_type => "log"
                }

Now when you add your xml messages into your log file. Each entry will be processed and stored in your elastic search server.

Thanks,

How to parse a xml-file with logstash filters

1 Answers1