403 error urllib crawling sites like google

Asked Jan 08 '15 at 13:44

Active Jan 08 '15 at 13:44

Viewed 235 times

I'm trying to crawl sites like google for a project and I keep on getting 403 errors. some sites work and others don't. none of the sites that i'm using have a apis other than google so thats out the window. any thoughts

FNLURL = "https://www.google.com/search?q=test"

try:
    x = urllib.request.urlopen(FNLURL)
    req = urllib.request.Request(FNLURL, headers={'User-Agent': 'Mozilla/5.0'})
    resp = urllib.request.urlopen(req)
    respData = resp.read()
    print(respData)

except Exception as t:
    print(str(t))

asked Jan 08 '15 at 13:44

Chanヨネ

1

possible duplicate of http://stackoverflow.com/questions/600536/why-does-google-search-return-http-error-403 – Paul Lo Jan 08 '15 at 13:49
I dont have a API to use on things other than google – Chanヨネ Jan 08 '15 at 13:50
In this case I suggest you to rotate proxies for each request you make – aberna Jan 08 '15 at 13:54
Doesn't seem to work – Chanヨネ Jan 08 '15 at 13:56

403 error urllib crawling sites like google

0 Answers0