1

I want to make a parser using "grequests", but as expected, due to many requests, i get an error 429.

import grequests


sites = [f"https://fantlab.ru/work{i}" for i in range(1, 10)]

response = (grequests.get(url, timeout=5)  for url in sites)
resp = grequests.map(response)

print(resp)

[<Response [200]>, <Response [200]>, <Response [200]>, <Response [404]>, <Response [429]>...]
Mop
  • 53
  • 5

1 Answers1

1

A 429 (Too many requests) status code is returned when you are sending too many requests. You could try showing down the requests (using time.sleep(n) from the time module, or see if the owner of the server can increase your quota.

Jaffa
  • 189
  • 1
  • 6
  • Okay, I understand your logic, but how should I use time.sleep and grequests at the same time, it's not requests. I tried doing it like this [code](http://pastie.org/p/6cpxiXQejSmcM74sW1m5aW) it works slower, but it doesn't solve the problem. – Mop Jul 18 '22 at 07:25
  • Sorry, I can't think of any other solutions other than asking the website owner – Jaffa Jul 18 '22 at 16:13