-1

i'm trying create a spider for scrawl a futbol site named futbin but i'm only can 10 sites, after 10 sites the error 429 aparrecing. P.S: i'm new to scrapy.

import scrapy


class FutbinSpider(scrapy.Spider):
    name = 'futbin'

    start_urls = [f'https://www.futbin.com/21/players?page={i}&sort=TotalStats&order=desc' for i in range(1, 21)]
    with open('items.json', 'w') as file:
        pass

    def parse(self, response):
        for indice_player in range(1, 31):
            yield {
                'name': response.xpath(f'//*[@id="repTb"]/tbody/tr[{indice_player}]/td[1]/div[2]/div[1]/a/text()').get(),
                'team': response.xpath(f'//*[@id="repTb"]/tbody/tr[{indice_player}]/td[1]/div[2]/div[2]/span/a[1]/@data-original-title').get(),
                'country': response.xpath(f'//*[@id="repTb"]/tbody/tr[{indice_player}]/td[1]/div[2]/div[2]/span/a[2]/@data-original-title').get(),
                'position': response.xpath(f'//*[@id="repTb"]/tbody/tr[{indice_player}]/td[3]/text()').get(),
                'base stats': response.xpath(f'//*[@id="repTb"]/tbody/tr[{indice_player}]/td[17]/text()').get()
            }```
Lenner Coutinho Pereira
  • 1
  • 1

1 Answers1

0

HTTP 429 Too Many Requests response status code indicates the user has sent too many requests in a given amount of time ("rate limiting"). Maybe the website tries to protect from web scrapers :-D

If you can allow waiting, then you can wait and try again. Maybe something like this How to handle a 429 Too Many Requests response in Scrapy?

David Miró
  • 2,694
  • 20
  • 20