0

I'm building a portal that lists certain products and automatically gets the prices from product pages of listed vendors. To get the URL for the product page on a vendor's website, I've been using Google search API and it's been working great - the first result is invariably the page of the product. However, now I'm getting errors saying that Google has blocked my website (actually my develoment machine's IP) from the API because I've been making automated requests such as scraping (the only item that applies).

Fine, Google can go jump off a cliff, but... how do product portals generally get URLs for thewir products? I can enter the URLs manually but that can be a problem if the vendor's website changes the URL scheme somehow. I obviously need an automated way to do this.

I'm making no more than 50-60 requests per day so I don't get what Google wants. Do they want money?

DMIL
  • 693
  • 3
  • 7
  • 18

1 Answers1

2

First, they want you to use one of their APIs, not scrape their web page directly. Their custom search API is documented here. Once you register they'll give you an API key. You can get results in JSON format by requesting

https://www.googleapis.com/customsearch/v1?q=SEARCH_TERMS&key=YOUR_KEY

Second, they do like money, but you might be okay. You're allowed 100 searches per day for free; beyond that you're you're going to be charged $5 per thousand searches.

Jeremy
  • 1
  • 85
  • 340
  • 366
  • I was already using their API (http://ajax.googleapis.com/ajax/services/search/web?v=1.0&q=whatever) but without the key. I was assuming I'm allowed those 100 searches without the API key. – DMIL Mar 27 '12 at 19:14
  • @DMIL Oh, I see. I think that that used to be fine, but Google is now trying to make people switch from their old API (which you are using) to the new one, so they're making the limits on the old one more strict! – Jeremy Mar 27 '12 at 19:23