1

I'm trying to scrape some information from Google Trends. But every time that I try to get some data I receive the error Too Many Requests. Other sites are ok.

My code:

func Teste(searchTrend string) {

    searchTrend = strings.Trim(searchTrend, " ")
    searchTrend = strings.Replace(searchTrend, " ", "%20", -1)

    linkTrends := ("https://trends.google.com.br/trends/explore?geo=BR&q=" + searchTrend)

    c := colly.NewCollector()

    c.SetRequestTimeout(120 * time.Second)

     c.OnRequest(func(r *colly.Request) {
        fmt.Println("Scraping:", r.URL)
     })

     c.OnResponse(func(r *colly.Response) {
        fmt.Println("Status:", r.StatusCode)
     })

    c.OnError(func(r *colly.Response, e error) {
        log.Println("error:", e, r.Request.URL, string(r.Body))
    })

    c.OnHTML("div.widget-template", func(h *colly.HTMLElement) {
      ...
    })

    c.Visit(linkTrends)

}

The error: That’s an error.

We're sorry, but you have sent too many requests to us recently. Please try again later. That’s all we know.

Maick Machado
  • 45
  • 6

1 Answers1

1

Your code seems good. However, Google usually blocks crawling activities from untrusted sources, that's why it returns 429 Too Many Requests.
You can find some tools out there that allow you to sign-up and crawl their trends. An example can be this.
To check the goodness of your solution, try to scrape another website and it should work as expected.

ossan
  • 1,665
  • 4
  • 10