Parse Document with Scrapy

Question

I have an issue, I want to parse a web site and crawl each article's links from it but the problem is Scrapy do not crawls all the links and crawls some of them a random number of times.

import scrapy

from tutorial.items import GouvItem

class GouvSpider(scrapy.Spider):

    name = "gouv"

    allowed_domains = ["legifrance.gouv.fr"]

    start_urls = [

        "http://www.legifrance.gouv.fr/affichCode.do?cidTexte=LEGITEXT000006069577&dateTexte=20160128"

        ]

    def parse(self, response):
        for href in response.xpath('//span/a/@href'):
            url = response.urljoin(href.extract())
            yield scrapy.Request(url, callback=self.parse_article)

    def parse_article(self, response):
        for art in response.xpath("//div[@class='corpsArt']"):
            item = GouvItem()
            item['article'] = art.xpath('p/text()').extract()
            yield item




#And this is the GouvItem :

import scrapy

class GouvItem(scrapy.Item):
    title1 = scrapy.Field()
    title2 = scrapy.Field()
    title3 = scrapy.Field()
    title4 = scrapy.Field()
    title5 = scrapy.Field()
    title6 = scrapy.Field()
    link = scrapy.Field()
    article = scrapy.Field()

This is some lines of the json file and we can see that some articles missing and others are there but many times

The problem is that each article of the law should be there and only one time. On the website, each article appears only time.

Thanks a lot !

Answer 1

The links to the sub-pages of the website contain a sessionID. It looks like the response for a request takes into account that sessionID in a manner that works not well with scrapy sending multiple concurrent requests.

One way to fix this is to set the number of CONCERRENT_REQUESTS in the settings.py to 1 . Scraping will take longer of course with this setting.

Another approach would be to control the the requests manually with a list. See this answer on SO.

To prevent empty results use a relative XPath (trailing dot) and extract all the text:

item['article'] = art.xpath('.//text()').extract()

Hope this helps.