0

I am trying to use google apps script's UrlFetchApp service to scrape top 5 google search results.

let response=UrlFetchApp.fetch("https://www.google.com/search?q="+encodeURIComponent("how to eat pizza" ),
{ "User-Agent": "Mozilla/5.0 (Linux; Android 7.0; SM-G930V Build/NRD90M) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/59.0.3071.125 Mobile Safari/537.36",
   "method"  : "GET",   
   }
).getContentText();

It was supposed to give similar results as searching through the browser.

I was expecting <div class="r"> (which will have href, title) in the response but don't see them.

When I send request using python's requests.get(URL, headers=headers) I was getting those tags <div class="r"> as expected.

I am unable to figure out the cause of the difference. Google custom search API is different I believe. urlfetchapp is supposed to give response for a simple get request.

user2688323
  • 15
  • 5
  • 2
    Does this answer your question? [How do I get Google search results from urlfetch in google apps script](https://stackoverflow.com/questions/29962902/how-do-i-get-google-search-results-from-urlfetch-in-google-apps-script) – dwmorrin Jul 11 '20 at 15:32
  • No..custom search works only when we specify which search engine to lookup. I am looking for a pure scrape get a similar result to hitting a url from the browser. It's quite possible in python or curl command.. – user2688323 Jul 11 '20 at 16:21
  • If you inspect the error HTML you get from Google you'll see that Google is smart and sees you trying to abuse their own services from Apps Script and that is why the approach you are taking cannot work. Using python or curl you are hitting Google from outside Google. Hitting Google with Google is different - you need to play by their rules. Does that make sense? – dwmorrin Jul 11 '20 at 16:25

0 Answers0