Python Scrapy - Yield statement not working as expected

Question

I have a Scrapy spider that looks like this. Basically it takes a list of URLs, follows the internal links and grabs the external links. What I'm trying to do is make it kind of synchronous so that the url_list is parsed in order.

class SomeSpider(Spider):
    name = 'grablinksync'
    url_list = ['http://www.sports.yahoo.com/', 'http://www.yellowpages.com/']
    allowed_domains = ['www.sports.yahoo.com', 'www.yellowpages.com']
    links_to_crawl = []
    parsed_links = 0

    def start_requests(self):
        # Initial request starts here
        start_url = self.url_list.pop(0)
        return [Request(start_url, callback=self.get_links_to_parse)]

    def get_links_to_parse(self, response):
        for link in LinkExtractor(allow=self.allowed_domains).extract_links(response):
            self.links_to_crawl.append(link.url)
            yield Request(link.url, callback=self.parse_obj, dont_filter=True)

    def start_next_request(self):
        self.parsed_links = 0
        self.links_to_crawl = []
        # All links have been parsed, now generate request for next URL
        if len(self.url_list) > 0:
            yield Request(self.url_list.pop(0), callback=self.get_links_to_parse)

    def parse_obj(self,response):
        self.parsed_links += 1
        for link in LinkExtractor(allow=(), deny=self.allowed_domains).extract_links(response):
            item = CrawlsItem()
            item['DomainName'] = get_domain(response.url)
            item['LinkToOtherDomain'] = link.url
            item['LinkFoundOn'] = response.url
            yield item
        if self.parsed_links == len(self.links_to_crawl):
            # This doesn't work
            self.start_next_request()

My problem is that the function start_next_request() is never called. If I move the code inside of start_next_request() inside the parse_obj() function, then it works as expected.

def parse_obj(self,response):
            self.parsed_links += 1
            for link in LinkExtractor(allow=(), deny=self.allowed_domains).extract_links(response):
                item = CrawlsItem()
                item['DomainName'] = get_domain(response.url)
                item['LinkToOtherDomain'] = link.url
                item['LinkFoundOn'] = response.url
                yield item
            if self.parsed_links == len(self.links_to_crawl):
                # This works..
                self.parsed_links = 0
                self.links_to_crawl = []
                # All links have been parsed, now generate request for next URL
                if len(self.url_list) > 0:
                    yield Request(self.url_list.pop(0), callback=self.get_links_to_parse)

I would like to abstract away the start_next_request() function because I'm planning on calling it from a few other places. I understand that it has something to do with start_next_request() being a generator function. But I'm new to generators and yields so I'm having a hard time figuring out what I did wrong.

Please study the posting guidelines more carefully, you are supposed to extract a minimal example. — Ulrich Eckhardt, Jan 05 '16 at 06:14

score 1 · Answer 1 · answered Jan 05 '16 at 10:26

That's because yield makes the function into a generator and simply writing self.start_next_request() doesn't make the generator do anything.

Generators are lazy, which means that unless you ask it for the first object - it wont do anything.

You can change the code to:

def parse_obj(self,response):
    self.parsed_links += 1
    for link in LinkExtractor(allow=(), deny=self.allowed_domains).extract_links(response):
        item = CrawlsItem()
        item['DomainName'] = get_domain(response.url)
        item['LinkToOtherDomain'] = link.url
        item['LinkFoundOn'] = response.url
        yield item
    if self.parsed_links == len(self.links_to_crawl):
        for res in self.start_next_request():
            yield res

Even return self.start_next_request() would work as you're returning the generator.

Python Scrapy - Yield statement not working as expected

1 Answers1

Linked