Log4j unstructured log parser in Java or Scala

Question

Question

How to parse the unstructured Log4J log in Java/Scala.

Background

There are multiple questions on how to parse Log4J logs but mostly they recommend using XML or JSON appender to output structured outputs to parse. However need to parse already generated unstructured log files programmatically, not using tools such as logstash, chainsaws GUI.

Help with LogFilePatternReceiver shows the way but it requires "plugin" support which was supposed to be implemented in Log4J 1.3 but Log4J 1.x discontinued with 1.2.x. It looks Log4J 2 does not have LogFilePatternReceiver.

<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE log4j:configuration >
<log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/" debug="true">
    <appender name="CONSOLE" class="org.apache.log4j.ConsoleAppender">
        <layout class="org.apache.log4j.SimpleLayout"/>
        <filter class="org.apache.log4j.filter.ExpressionFilter">
            <param name="Expression" value="msg ~= 'SPECIFY CASE-INSENSITIVE MSG MATCH HERE'" />
            <param name="AcceptOnMatch" value="true"/>
        </filter>
        <filter class="org.apache.log4j.filter.DenyAllFilter"/>
    </appender>

    <plugin name="LOGFILE" class="org.apache.log4j.varia.LogFilePatternReceiver">
        <param name="fileURL" value="file:///c:/some.log"/>
        <param name="timestampFormat" value="yyyy-MM-dd HH:mm:ss,SSS"/>
        <param name="logFormat" value="TIMESTAMP LEVEL [LOGGER] MESSAGE"/>
        <param name="name" value="my log file"/>
        <param name="tailing" value="true"/>
    </plugin>

    <root>
        <level value="DEBUG"/>
        <appender-ref ref="CONSOLE"/>
    </root>
</log4j:configuration>

There is a class LogFileParser in org.apache.log4j.lf5.util.LogFileParse package but not sure how to use it. There is a tool OtrosLogViewer but prefer not having 3rd party dependencies if possible.

Options

Re: Extract log info from each output log line on the console suggests refering to the Chainsaw code. Read the code and implement similar would be the option but if there is already a known way to parse logs, would like to know it first.

References

mon · Accepted Answer · 2018-02-26T12:12:29.997

Found a way by using log4j and extras 1.2 jars, although not work with Log4J 2.

Java

import org.apache.log4j.Logger;
import org.apache.log4j.spi.LoggingEvent;
import org.apache.log4j.receivers.varia.LogFilePatternReceiver;

public class LogParser extends LogFilePatternReceiver {
    final static Logger logger = Logger.getLogger(LogParser.class);

    public void doPost(LoggingEvent event) {
        System.out.println(event.getTimeStamp());
        System.out.println(event.getMessage());
    }
    public static void main(String[] args) {
        (new LogParser()).run();
    }
    private void run(){
        LogFilePatternReceiver r = new LogParser();
        r.setLogFormat("TIMESTAMP LEVEL [THREAD] CLASS (FILE:LINE) - MESSAGE");
        r.setFileURL("file:///tmp/log");
        r.setTimestampFormat("yyyy-MM-dd HH:mm:ss,SSS");
        r.setTailing(true);
        r.setLoggerRepository(logger.getLoggerRepository());

        r.activateOptions();
    }
}

log4j.xml

<?xml version="1.0" encoding="UTF-8" ?>
<!DOCTYPE log4j:configuration SYSTEM "log4j.dtd">

<log4j:configuration xmlns:log4j="http://jakarta.apache.org/log4j/">
    <appender name="console" class="org.apache.log4j.ConsoleAppender">
        <param name="Target" value="System.out"/>
        <layout class="org.apache.log4j.PatternLayout">
            <param name="ConversionPattern" value="%d %-5p [%t] %C{2} (%F:%L) - %m%n"/>
        </layout>
    </appender>
    <root>
        <priority value ="debug" />
        <appender-ref ref="console" />
    </root>
</log4j:configuration>

Sample log file

2017-12-02 16:53:42,618 DEBUG [main] LogParser (LogParser.java:15) - message 1
2017-12-02 16:53:42,624 DEBUG [main] LogParser (LogParser.java:16) - message 2

Result

2017-12-02 19:21:31,633 INFO  [main] spi.Log4JULogger (Log4JULogger.java:104) - activateOptions
2017-12-02 19:21:31,652 DEBUG [LogFilePatternReceiver-plugin] spi.Log4JULogger (Log4JULogger.java:58) - regexp is (\S+-\S+-\S+ \S+:\S+:\S+,\S+)[ ]+(\S*\s*?)[ ]+\[(.*?)\][ ]+(.*?)[ ]+\((.*?):(.*?)\)[ ]+\-[ ]+(.*)
2017-12-02 19:21:31,652 INFO  [LogFilePatternReceiver-plugin] spi.Log4JULogger (Log4JULogger.java:104) - attempting to load file: file:///tmp/log

1512194022618
message 1
1512194022624
message 2

2017-12-02 19:21:33,656 DEBUG [LogFilePatternReceiver-plugin] spi.Log4JULogger (Log4JULogger.java:58) - tailing file
2017-12-02 19:21:35,660 DEBUG [LogFilePatternReceiver-plugin] spi.Log4JULogger (Log4JULogger.java:58) - tailing file

Log4j unstructured log parser in Java or Scala

Question

Background

Options

References

1 Answers1