I am trying out some code which works great when I only have ~50 requests to send, but now I'm trying with more requests (about 200) and I feel like the server might be closing the connection after 60 requests, because I get "None" responses.
Here's the code I'm using:
urls = []
for kw in keywords:
urls.append("https://play.google.com/store/search?q=" + kw + "&c=apps&hl=fr")
from lxml import html
rs = (grequests.get(u) for u in urls)
responses = grequests.map(rs)
for idx, r in enumerate(responses): # r = results page
time.sleep(2)
responses_tree = html.fromstring(r.content) # Readable HTML
responses:
Out[893]:
[<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
<Response [200]>,
None,
<Response [200]>,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None,
None]
and the traceback:
AttributeError Traceback (most recent call last)
C:\Users\Nathan\googlestore.py in <module>()
249 for idx, r in enumerate(responses): # r = results page
250 time.sleep(2)
--> 251 responses_tree = html.fromstring(r.content) # Readable HTML
252 apps_links = responses_tree.xpath('//a[@class="title"]/@href') # Get all links
253 if len(apps_links) > 10:
AttributeError: 'NoneType' object has no attribute 'content'
How can I fix this? Should I wait like 30 seconds after 50 requests have been sent, then send another 50? Does anybody have a link to Google documentation about requests made to their servers?
Thanks!