1

I am trying out some code which works great when I only have ~50 requests to send, but now I'm trying with more requests (about 200) and I feel like the server might be closing the connection after 60 requests, because I get "None" responses.

Here's the code I'm using:

urls = []
for kw in keywords:
    urls.append("https://play.google.com/store/search?q=" + kw + "&c=apps&hl=fr")

from lxml import html

rs = (grequests.get(u) for u in urls)
responses = grequests.map(rs)

for idx, r in enumerate(responses): # r = results page
    time.sleep(2)
    responses_tree = html.fromstring(r.content) # Readable HTML

responses:

Out[893]: 
[<Response [200]>,
 <Response [200]>,
 <Response [200]>,
 <Response [200]>,
 <Response [200]>,
 <Response [200]>,
 <Response [200]>,
 <Response [200]>,
 <Response [200]>,
 <Response [200]>,
 <Response [200]>,
 <Response [200]>,
 <Response [200]>,
 <Response [200]>,
 <Response [200]>,
 <Response [200]>,
 <Response [200]>,
 <Response [200]>,
 <Response [200]>,
 <Response [200]>,
 <Response [200]>,
 <Response [200]>,
 <Response [200]>,
 <Response [200]>,
 <Response [200]>,
 <Response [200]>,
 <Response [200]>,
 <Response [200]>,
 <Response [200]>,
 <Response [200]>,
 <Response [200]>,
 <Response [200]>,
 <Response [200]>,
 <Response [200]>,
 <Response [200]>,
 <Response [200]>,
 <Response [200]>,
 <Response [200]>,
 <Response [200]>,
 <Response [200]>,
 <Response [200]>,
 <Response [200]>,
 <Response [200]>,
 <Response [200]>,
 <Response [200]>,
 <Response [200]>,
 <Response [200]>,
 <Response [200]>,
 <Response [200]>,
 <Response [200]>,
 <Response [200]>,
 <Response [200]>,
 <Response [200]>,
 <Response [200]>,
 <Response [200]>,
 <Response [200]>,
 <Response [200]>,
 <Response [200]>,
 <Response [200]>,
 <Response [200]>,
 <Response [200]>,
 None,
 <Response [200]>,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None,
 None]

and the traceback:

AttributeError                            Traceback (most recent call last)
C:\Users\Nathan\googlestore.py in <module>()
    249 for idx, r in enumerate(responses): # r = results page
    250     time.sleep(2)
--> 251     responses_tree = html.fromstring(r.content) # Readable HTML
    252     apps_links = responses_tree.xpath('//a[@class="title"]/@href') # Get all links
    253     if len(apps_links) > 10:

AttributeError: 'NoneType' object has no attribute 'content' 

How can I fix this? Should I wait like 30 seconds after 50 requests have been sent, then send another 50? Does anybody have a link to Google documentation about requests made to their servers?

Thanks!

vandernath
  • 3,665
  • 3
  • 15
  • 24

0 Answers0