How to pattern multiline XML using RegEx in NXLog

Question

I am trying to parse a custom log file into JSON using the nxLog parser to_json() so that I can then ship them into my ElasticSearch instance. I am going to be splitting these into three separate fields, date, log type indicator, and message.

Following is the format of these logs.

9/10/2015 11:30:05 AM [0-1-1-Pos.xaml.cs-1607] Post button clicked

9/10/2015 11:30:17 AM [0-3-1-SecondaryPortStatus.cs-47] <TRANSACTION>
  <FUNCTION_TYPE>SECONDARYPORT</FUNCTION_TYPE>
  <COMMAND>STATUS</COMMAND>
  <MAC_LABEL>XX</MAC_LABEL>
  <MAC>xOel7QeyKoXaddiyrEeWKRI1DlF9sHzUNfZHFI/gAko=</MAC>
 <COUNTER>XXX</COUNTER>
</TRANSACTION>

9/10/2015 11:30:17 AM [0-3-1-SecondaryPortStatus.cs-57] <RESPONSE>
  <RESPONSE_TEXT>Operation SUCCESSFUL</RESPONSE_TEXT>
  <RESULT>OK</RESULT>
  <RESULT_CODE>-1</RESULT_CODE>
  <TERMINATION_STATUS>SUCCESS</TERMINATION_STATUS>
  <COUNTER>221</COUNTER>
  <SECONDARY_DATA>12</SECONDARY_DATA>
  <MACLABEL_IN_SESSION>P_061</MACLABEL_IN_SESSION>
  <SESSION_DURATION>00:00:16</SESSION_DURATION>
  <INVOICE_SESSION>XX</INVOICE_SESSION>
  <SERIAL_NUMBER>XX</SERIAL_NUMBER>
</RESPONSE>`

I've been able to parse date stamp and the error selector (everything within brackets) using PERL regex syntax as follows.

1. ^(\d\d|\d)/(\d\d|\d)/(\d\d\d\d)\s(\d\d|\d):(\d\d|\d):(\d\d|\d)\s(AM|PM) 
2. \[(.*)\]

Date
Log Type Identifier
Message which will be what I am trying to figure out.

But I cannot figure out how to pull everything between the selector and the new line. So in this instance I'd like my message to be the XML code until the newline. Does anyone have advice on how I can retrieve the data?

use the flip-flop operator: http://perldoc.perl.org/perlop.html#Range-Operators or http://perlhacks.com/2014/01/dots-perl/ — Casimir et Hippolyte, Sep 14 '15 at 21:27

score 1 · Accepted Answer · edited Sep 15 '15 at 16:37

1

You should be able to use nxlog's xm_multiline module and specify the regexp in the HeaderLine directive. If you add a capturing rule to the regexp to match the XML part (stuff after the [..]) then you should be able to parse the XML with xm_xml's parse_xml().

There is a similar example here.

edited Sep 15 '15 at 16:37

C Ried

290
4
15

answered Sep 15 '15 at 13:52

b0ti

2,319
1
18
18

Gilles Quénot · Answer 2 · 2015-09-14T23:45:15.797

Try doing this using a multiline ReGex :

$ perl -0777 -ne 'print $& if !<RESPONSE>.*</RESPONSE>!s' file

Setting the input/output separator as undef (-0777) will slurp the whole file in memory

Output:

<RESPONSE>
  <RESPONSE_TEXT>Operation SUCCESSFUL</RESPONSE_TEXT>
  <RESULT>OK</RESULT>
  <RESULT_CODE>-1</RESULT_CODE>
  <TERMINATION_STATUS>SUCCESS</TERMINATION_STATUS>
  <COUNTER>221</COUNTER>
  <SECONDARY_DATA>12</SECONDARY_DATA>
  <MACLABEL_IN_SESSION>P_061</MACLABEL_IN_SESSION>
  <SESSION_DURATION>00:00:16</SESSION_DURATION>
  <INVOICE_SESSION>XX</INVOICE_SESSION>
  <SERIAL_NUMBER>XX</SERIAL_NUMBER>
</RESPONSE>

In a script :

BEGIN { $/ = undef; $\ = undef; } # input/output separator as undef
while (defined($_ = <ARGV>)) {
    print $& if m[<RESPONSE>.*</RESPONSE>]s;
}

from `perldoc perlre` for modifier 's'

 s   Treat string as single line. That is, change "." to match any
     character whatsoever, even a newline, which normally it would not
     match.

How to pattern multiline XML using RegEx in NXLog

2 Answers2

Output:

In a script :

from perldoc perlre for modifier 's'

from `perldoc perlre` for modifier 's'