Parsing log4j log files with multi-line entries (i.e. Java Stack Exceptions)

Question

How do I use regex to parse a log4j 1 log file that can contain multiple lines per event? (i.e. java exception stack traces)

Sample log4j log output

Pattern layout %r [%t] %-5p %c{2} %x - %m%n

176 [main] INFO  examples.Sort - Populating an array of 2 elements in reverse order.
225 [main] INFO  examples.SortAlgo - Entered the sort method.
262 [main] DEBUG SortAlgo.OUTER i=1 - Outer loop.
276 [main] DEBUG SortAlgo.SWAP i=1 j=0 - Swapping intArray[0] = 1 and intArray[1] = 0
290 [main] DEBUG SortAlgo.OUTER i=0 - Outer loop.
304 [main] INFO  SortAlgo.DUMP - Dump of interger array:
317 [main] INFO  SortAlgo.DUMP - Element [0] = 0
331 [main] INFO  SortAlgo.DUMP - Element [1] = 1
343 [main] INFO  examples.Sort - The next log statement should be an error message.
346 [main] ERROR SortAlgo.DUMP - Tried to dump an uninitialized array.
        at org.log4j.examples.SortAlgo.dump(SortAlgo.java:58)
        at org.log4j.examples.Sort.main(Sort.java:64)
467 [main] INFO  examples.Sort - Exiting main method. <-- regex fails to capture last event

This regex fails to capture the last event

flags: /gsmx

^(?<elapsed>\d+?) \s 
\[(?<thread>.+?)\] \s 
(?<priority>TRACE|DEBUG|INFO|WARN|ERROR|FATAL) \s 
(?<category>.+?) \s \- \s 
(?<msg>
  (.+?$[\n])+?
    (
      $(?![\r\n])(?#EOF)
      |
      (?=\d+? \s \[ .+? \] \s (TRACE|DEBUG|INFO|WARN|ERROR|FATAL) \s (.+?) \s \-)
  )
)

I know log4j 2 can encode but I have not found a way to swap out to the new version yet.

score 0 · Answer 1 · answered Jan 13 '17 at 04:00

Finally nailed the regex:

(?smx)
^(?<elapsed>\d+?) \s 
\[(?<thread>.+?)\] \s 
(?<priority>TRACE|DEBUG|INFO|WARN|ERROR|FATAL) \s 
(?<category>.+?) \s \- \s 
(?<msg>
  (.+?)
    (
    $(?![\r\n]) (?#EOF)
    |
    (?=^\d+? \s \[ .+? \] \s (TRACE|DEBUG|INFO|WARN|ERROR|FATAL) \s (.+?) \s \-)
  )
)

Parsing log4j log files with multi-line entries (i.e. Java Stack Exceptions)

1 Answers1