scrapy allow all domains

Question

I saw this post to make scrapy crawl any site without allowed domains restriction.

Is there any better way of doing it, such as using a regular expression in allowed domains variable, like-

allowed_domains = ["*"]

I hope there is some other way than hacking into scrapy framework to do this.

score 14 · Accepted Answer · answered Mar 03 '12 at 04:29

14

Don't set allowed_domains at all.

Look at the get_host_regex() function in this scrapy file:

answered Mar 03 '12 at 04:29

Shawn Lewis

Thanks. But, I want to set allowed_domains to all, is that possible? – hrishikeshp19 Mar 04 '12 at 04:22
10

If you don't set allowed_domains then all domains are allowed by default. – Shawn Lewis Mar 04 '12 at 18:01
I have a class that extends scrapy spider and then another class that describes that one. So I have allowed domains in that intermediate class, but want to remove them from the final class – Karina Klinkevičiūtė Jan 20 '21 at 11:15

score 1 · Answer 2 · answered Aug 24 '13 at 23:56

1

you should diactivate offsite middlware which is a built in spider middleware in scrapy. for more information http://doc.scrapy.org/en/latest/topics/spider-middleware.html

answered Aug 24 '13 at 23:56

Jhon Garside

2 Answers2