I'm crawling this website using grequests (at a pretty slow pace), and at the first few hundreds of requests I get mostly 200 HTTP responses, but at a certain point, all I get are None responses(without any raised exception).
The weirdest part is the if I try running the crawl again while the first crawl is still running and getting None responses, the second crawl received 200 responses.
By the way, I'm pretty sure the website I'm crawling doesn't have any kind of crawling protection.
This is my code:
for x in range(0, len(urls), JUMP):
rs = (grequests.get(u, stream=False, headers = headers,
timeout = 300) for u in urls[x:x+JUMP])
responses += grequests.map(rs,exception_handler=nones_exception_handler)
# saves the failed requests so i can try those again later
def nones_exception_handler(request, exception):
nonlocal exceptionurls
print(exception)
exceptionurls.append(request.url)
