How to prevent logging of pyspark 'answer received' and 'command to send' messages

Question

I am using python logging with pyspark and pyspark DEBUG level messages are flooding my log file with the example shown. How do I prevent this from happening? A simple solution is to set log level to INFO, but I need to log my own python DEBUG level messages

2015-12-13 15:13:32 4906 DEBUG   : Command to send: j
i
rj
org.apache.spark.SparkConf
e

2015-12-13 15:13:32 4906 DEBUG   : Answer received: yv
2015-12-13 15:13:32 4906 DEBUG   : Command to send: j
i
rj
org.apache.spark.api.java.*
e

2015-12-13 15:13:32 4906 DEBUG   : Answer received: yv
2015-12-13 15:13:32 4906 DEBUG   : Command to send: j
i
rj
org.apache.spark.api.python.*
e

score 6 · Accepted Answer · answered Dec 13 '15 at 07:37

6

You can set logging level for each logger separately

 pyspark_log = logging.getLogger('pyspark')
 pyspark_log.setLevel(logging.ERROR)

answered Dec 13 '15 at 07:37

Andy

161
3

score 3 · Answer 2 · answered Oct 24 '18 at 06:50

I had the same issue, I used following and all worked fine.

pyspark_log = logging.getLogger('pyspark').setLevel(logging.ERROR)
py4j_logger = logging.getLogger("py4j").setLevel(logging.ERROR)
matplotlib_logger = logging.getLogger("matplotlib").setLevel(logging.ERROR)

I was getting some matplotlib lib logs also, so I changed matplotlib logger level too, but if you don't have that issue you can remove that line.

Z.Wei · Answer 3 · 2021-01-07T20:01:55.707

3

The key component is "py4j". You just need to add a line of code to the beginning of your program:

py4j_logger = logging.getLogger("py4j").setLevel(logging.INFO)

Or just:

logging.getLogger("py4j").setLevel(logging.INFO)

edited Jan 07 '21 at 20:01

answered Jul 12 '19 at 13:44

Z.Wei

3,658
2
17
28

score 1 · Answer 4 · answered Oct 20 '17 at 15:46

1

logging.basicConfig(level=logging.DEBUG)
logging.getLogger('py4j').setLevel(logging.INFO) # use setLevel(logging.ERROR) is also fine
logging.getLogger('pyspark')

logging.info('Task is successful.')

answered Oct 20 '17 at 15:46

Rong Du

291
4
3

score 0 · Answer 5 · answered May 17 '21 at 11:31

The best way to control pyspark and py4j logging is by setting the following snippet:

import logging
logging.getLogger("py4j").setLevel(<pyspark-level>)
logging.getLogger('pyspark').setLevel(<py4j-level>)
logger = logging.getLogger('pyspark')

For your case you should write:

import logging
logging.getLogger("py4j").setLevel(logging.DEBUG)
logging.getLogger('pyspark').setLevel(logging.WARNING)
logger = logging.getLogger('pyspark')

How to prevent logging of pyspark 'answer received' and 'command to send' messages

5 Answers5