Nested rules for CrawlSpider in Scrapy

Asked May 25 '18 at 05:03

Active May 25 '18 at 05:03

Viewed 161 times

Totally new to scrapy and crawlspider.. I'm stuck on how to define rules for nested crawling?? I've a rule defined as

Rule(LinkExtractor(
            allow=(),
            restrict_xpaths='//div[@class="sch-main-menu-sub-links-left"]'
            ), callback='parse_item', follow=True)

It crawls categories page, then, from each category page extracted from this rule, I've to crawl all the products on that page with rule

Rule(LinkExtractor(
            allow=(),
            restrict_xpaths='div[@class="sch-category-products-item"]'
            ), callback='parse_product', follow=True)

Also the category page is paginated.. But I'm not getting on how to do this. first rule is successful but where to place second Rule? Is there any way in CrawlSpider to define rule levels??

asked May 25 '18 at 05:03

F. Shahid

Possible duplicate of [How do Scrapy rules work with crawl spider](https://stackoverflow.com/questions/22082938/how-do-scrapy-rules-work-with-crawl-spider) – Andrés Pérez-Albela H. May 25 '18 at 05:24
Problem is almost same but none of the answers is relevant... – F. Shahid May 25 '18 at 05:37

Nested rules for CrawlSpider in Scrapy

0 Answers0