Regular expression for nxlog

Question

I have a set of logs which do not have a fixed line number. I am trying to get these logs to Elasticsearch through nxlog.

     ------------------------------------------------------
    Timestamp: 2020-03-03 00:05:04Z|User -DEFAULT|Cart -DEFAULT|Source -mscorlib|Message -StartIndex cannot be less than zero
     ------------------------------------------------------
     ------------------------------------------------------    
    Timestamp: 2020-03-03 03:05:53Z|User -Simulate Pricing | Initiate | |Cart -|Level -INFO|Severity -Information|Class -SalesOrderClient.vb|Function - Simulate(testVal As Object) Method :| Cart_ID: 8092662

    Timestamp: 2020-03-03 00:00:30Z|User -|Cart -|Source -mscorlib|Message -Thread was being aborted.|StackTrace - at System.Threading.Thread.AbortInternal()
        at System.ServiceModel.Channels.RequestChannel.Request(Message message, TimeSpan timeout)
        at System.ServiceModel.Dispatcher.RequestChannelBinder.Request(Message message, TimeSpan timeout)
        at System.ServiceModel.Channels.HttpChannelFactory`1.HttpRequestChannel.HttpChannelRequest.SendRequest(Message message, TimeSpan timeout)
        at System.ServiceModel.Channels.ServiceChannelProxy.Invoke(IMessage message)
     ------------------------------------------------------
     ------------------------------------------------------

    Timestamp: 2020-03-03 00:18:57Z|User -DEFAULT|Cart -DEFAULT|Source -mscorlib|Message -StartIndex cannot be less than zero.

     ------------------------------------------------------

This has a few logs with multiline (no fixedlines) and a few with a single line. When im trying to fetch this data with nxlog, my third log is getting split into 4 different rows for each new line present.

How can I avoid this issue. What Regular expression can I use here? I want one full log to be in a single row, irrespective of how many line there are. The lines are present in the logs.

Thanks in Advance.

score 2 · Accepted Answer · answered Mar 04 '20 at 22:27

2

Use a regexp for HeaderLine with the xm_multiline module that matches your timestamp.
See the examples at https://nxlog.co/documentation/nxlog-user-guide/xm_multiline.html

answered Mar 04 '20 at 22:27

b0ti

2,319
1
18
18

Thank you, Yes I did try it, except I'm not able to avoid the Dashes (------------------------------------) in my lines. Anyway to avoid that? – AlisonGrey Mar 05 '20 at 04:28
Yes you can get rid of that by adding this to your `xm_multiline` instance: `Exec if $raw_event =~ /^----/ drop();` – b0ti Mar 06 '20 at 10:44
It would drop the whole message since it is in multi line, however its okay. Thanks! – AlisonGrey Mar 06 '20 at 12:05
1

No, it only drops the line with the dashes if you add this under `xm_multiline`. There is a similar example in the user guide. If you want to get rid of the dashes and keep the line then you can rewrite it as well. – b0ti Mar 07 '20 at 13:24

Regular expression for nxlog

1 Answers1