How To Turn Off Logging in Scrapy (Python)

Question

I have created a spider using Scrapy but I cannot figure out how to turn off the default logging. From the documentation it appears that I should be able to turn it off by doing

        logging.basicConfig(level=logging.ERROR)

But this has no effect. From looking at the code for logging.basicConfig() I'm guessing this is because "the root logger has handlers configured" but perhaps I'm wrong about that. At any rate, can anyone explain what I need to do to get Scrapy to not output the usual

        2015-10-18 17:42:00 [scrapy] INFO: Scrapy 1.0.3 started (bot: EF)
        2015-10-18 17:42:00 [scrapy] INFO: Scrapy 1.0.3 started (bot: EF)
        2015-10-18 17:42:00 [scrapy] INFO: Optional features available: ssl, http11, boto
        2015-10-18 17:42:00 [scrapy] INFO: Optional features available: ssl, http11, boto

etc.?

EDIT: As suggested by sirfz below, the line

        logging.getLogger('scrapy').setLevel(logging.WARNING)

can be used to set the logging level. However, it appears that you must do this in the init method (or later) in your spider.

score 31 · Accepted Answer · answered Oct 19 '15 at 00:05

31

You can simply change the logging level for scrapy (or any other logger):

logging.getLogger('scrapy').setLevel(logging.WARNING)

This disables all log messages less than the WARNING level.

To disable all scrapy log messages you can just set propagate to False:

logging.getLogger('scrapy').propagate = False

This prevents scrapy's log messages from propagating to the root logger (which prints to console when configured using basicConfig())

answered Oct 19 '15 at 00:05

sirfz

4,097
23
37

4

Thank you for your help! I've figured out that setting the level works only if I put it in the __init__ of my spider. I'm running my spider from a script -- if I set the logging level before I do start the crawl it (apparently) gets reset to INFO. I'm not explicitly calling "configure_logging" but I guess Scrapy is doing that somewhere during startup. (?) – Dr. Pain Oct 19 '15 at 02:39
I never used scrapy myself but I guess the crawler is applying a default configuration for `logging` which is weird as this should be left for the user. – sirfz Oct 19 '15 at 18:06

score 7 · Answer 2 · edited Sep 17 '16 at 05:55

7

You could add -s LOG_ENABLED=False as a parameter when launching your script. That should do the trick.

Note: For the version 1.1 changed a little bit: -s LOG_ENABLED=0

edited Sep 17 '16 at 05:55

omar

1,541
3
21
36

answered Oct 19 '15 at 00:00

Steven_VdB

108
1
7

1

that works, but it's a bit radical, as it turns off all logging, including errors. You might try `-s LOG_LEVEL=WARNING` or another of the 5 log levels described here: https://docs.scrapy.org/en/latest/topics/logging.html#log-levels – Kai Carver Oct 02 '20 at 12:30

score 3 · Answer 3 · answered Sep 11 '19 at 18:54

3

you can simply add --nolog as a parameter when launching your spider using scrapy command
I am using scrapy v1.7.3. you can see more in help using command:

scrapy --help

answered Sep 11 '19 at 18:54

Siddhant

41
5

score 1 · Answer 4 · answered Mar 20 '17 at 12:31

1

logging.basicConfig(**kwargs)

This function does nothing if the root logger already has handlers configured for it.

Scrapy has handlers configured for it, so this will not work

answered Mar 20 '17 at 12:31

宏杰李

11,820
2
28
35

Accodius · Answer 5 · 2022-08-10T19:42:04.120

1

This might helpful, mentioned in the document that you can add this to setting file:

LOG_LEVEL = 'WARNING'

edited Aug 10 '22 at 19:42

answered Aug 10 '22 at 19:41

Accodius

13
1
3

How To Turn Off Logging in Scrapy (Python)

5 Answers5