0

I recently began to try to learn web scraping using Scrapy. Recently I tried to Scrapycrawl through the books.toscrape.com. According to the terminal, the Scrapycrawl call works fine, but it doesn't return the item count nor does it show any of the information on the books. I was wondering if this was normal, or if I have an error in my code.

Here is my code for the problem if this helps anyone understand what I was doing.

import scrapy

class BookspiderSpider(scrapy.Spider):
    name = "bookspider"
    allowed_domains = ["books.toscrape.com"]
    start_urls = ["https://books.toscrape.com"]


def parse(self, response):
    books = response.css('article.product_pod')
    for book in books:
    yield{
        'name': book.css('h3 a::text').get(),
        'url': book.css('h3 a').attrib['href'],
           }
        
    next_page = response.css('li.next a ::attr(href)').get()
    if next_page is not None:
        if 'catalogue/' in next_page:
             next_page_url = 'https://books.toscrape.com/' + next_page
        else:
            next_page_url = 'https://books.toscrape.com/catalogue/' + next_page 
        yield response.follow(next_page_url , callback = self.parse)
Byte Ninja
  • 881
  • 5
  • 13
Holden Eagle
  • 1
  • well your indentation is wrong, but that might just be a copy and paste issue. Otherwise your code seems fine and runs fine on my end. It does produce the correct output including the count and extracted details. – Alexander Aug 22 '23 at 01:23

0 Answers0