log4j.properties file not found on classpath or ignored

Question

I want to log in maprDB a spark job with log4j. I have written a custom appender, and here my log4j.properties :

log4j.rootLogger=INFO, stdout

log4j.appender.stdout=org.apache.log4j.ConsoleAppender log4j.appender.stdout.Target=System.out log4j.appender.stdout.layout=org.apache.log4j.PatternLayout log4j.appender.stdout.layout.ConversionPattern=%d{yyyy-MM-dd HH:mm:ss} %-5p %c{1}:%L - %m%n

log4j.appender.MapRDB=com.datalake.maprdblogger.Appender

log4j.logger.testest=WARN, MapRDB

Put on src/main/resources directory

This is my main method :

object App {
    val log: Logger = org.apache.log4j.LogManager.getLogger(getClass.getName)

    def main(args: Array[String]): Unit = {
        // custom appender
        LogHelper.fillLoggerContext("dev", "test", "test", "testest", "")
        log.error("bad record.")
    }
}

When I run my spark-submit without any configuration, nothing happens. It is like my log4j.properties wasn't here.

If I deploy my log4j.properties file manually and add options :

--conf spark.driver.extraJavaOptions=-Dlog4j.configuration=file:/PATH_TO_FILE/log4j.properties

--conf spark.executor.extraJavaOptions=-Dlog4j.configuration=file:/PATH_TO_FILE/log4j.properties

It works well. Why it doesn't work without theese options ?

What build tool do you use? Shouldn't the path be `src/main/resources` instead? — Jacek Laskowski, Dec 21 '17 at 14:35
I also think `log4j.appender.stdout.Target` (uppercase) should be `log4j.appender.stdout.target` (lowercase). — Jacek Laskowski, Dec 21 '17 at 14:36
The log4j.properties works well if I set the path explicitly. The problem is spark and his log4j.properties which is selected in first. — Franck Cussac, Dec 21 '17 at 16:15
Use `-Dlog4j.debug` and see where log4j searches for the properties. — Jacek Laskowski, Dec 21 '17 at 18:50

score 0 · Answer 1 · answered Jul 25 '19 at 07:15

The "spark.driver.extraJavaOptions" :

Default value is: (none)

A string of extra JVM options to pass to the driver. For instance, GC settings or other logging. Note that it is illegal to set maximum heap size (-Xmx) settings with this option. Maximum heap size settings can be set with spark.driver.memory in the cluster mode and through the --driver-memory command-line option in the client mode.

Note: In client mode, this config must not be set through the SparkConf directly in your application, because the driver JVM has already started at that point. Instead, please set this through the --driver-java-options command line option or in your default properties file.

[Refer to this link for more details: https://spark.apache.org/docs/latest/configuration.html]

log4j.properties file not found on classpath or ignored

1 Answers1