Excessive parquet messages on Spark 2.1

Question

After trying every approach shown on this question: How to suppress parquet log messages in Spark? none works with Spark 2.1 - except the blunt instrument approach of disabling all logging below WARN level.

log4j.rootCategory=WARN, console

That is not an acceptable approach (our app writes INFO messages for reason .. ).

Note the first approach taken was to add

log4j.logger.parquet=ERROR
log4j.logger.org.apache.spark.sql.execution.datasources.parquet=ERROR
log4j.logger.org.apache.spark.sql.execution.datasources.FileScanRDD=ERROR
log4j.logger.org.apache.hadoop.io.compress.CodecPool=ERROR

to the log4j.properties. These had no effect. The other approach is also included in my attempts:

org.apache.parquet.handlers=java.util.logging.ConsoleHandler
java.util.logging.ConsoleHandler.level=SEVERE

with the following added to the jvm options

 -Dspark.driver.extraJavaOptions="-Djava.util.logging.config.file=/tmp/parquet.logging.properties"
  -Dspark.executor.extraJavaOptions="-Djava.util.logging.config.file=/tmp/parquet.logging.properties"

Likewise no change.

If anyone has found a magic Quiet down Parquet button please chime in.

score 0 · Answer 1 · answered Jul 26 '17 at 06:25

0

Add:

log4j.logger.org.apache.parquet=ERROR
log4j.logger.parquet=ERROR

to your log4j.properties file

answered Jul 26 '17 at 06:25

wllmtrng

182
3

Thx. That had been the first thing attempted (and which did not have any effect). I made that more explicit in the OP. – WestCoastProjects Jul 26 '17 at 14:28

Excessive parquet messages on Spark 2.1

1 Answers1