What does Scrapy Job Setting mean?

Question

I was reading https://doc.scrapinghub.com/scrapy-cloud.html#scrapycloud , and kind of confused what does it mean to override a Scrapy settings for a job. Does it mean that I can change the start_url? Or which setting that I can really override.

Sorry for my bad English.

the settings that are in the file settings.py inside your spider, you can find a list in the documentation page https://doc.scrapy.org/en/latest/topics/settings.html — Mauricio Cortazar, Apr 26 '18 at 20:57
@MauricioCortazar can I add my own custom setting? like I want to add LIST_URLs as my start_url so I can overwrite it later using API. — Andre Rumapea, Apr 26 '18 at 21:38
check this https://doc.scrapy.org/en/latest/topics/settings.html#settings-per-spider — Mauricio Cortazar, Apr 26 '18 at 21:40
and according with the message from the page it can be done `job_settings should be a valid JSON and will be merged with project and spider settings provided for given spider.` — Mauricio Cortazar, Apr 26 '18 at 21:41
^ this should be the answer. To expand on this as far as I understand these settings will be only applied to current job, same as you'd use `-s` flags with your `scrapy crawl` commands for individual runs - here you'd use `job_settings` to provide multiple settings in json format. — Granitosaurus, Apr 27 '18 at 04:11
@MauricioCortazar you can add last comment as the answer. I will accept it as an answer. Thanks! — Andre Rumapea, May 24 '18 at 07:13

score 0 · Accepted Answer · answered May 24 '18 at 14:04

0

According with the example provided in the page you mention, you can do it:

job_settings should be a valid JSON and will be merged with project and spider settings provided for given spider

answered May 24 '18 at 14:04

Mauricio Cortazar

4,049
2
17
27

What does Scrapy Job Setting mean?

1 Answers1