YQL "Redirected to a robots.txt restricted URL" Error for Google Domain

Question

I am using YQL Console and I want to return results from this link in Google Shopping

Using the following in YQL:

  select content from html where url="www.google.com/products/seller?hl=en&q=burkett+restaurant+equipment+reviews&tbo=u&tbm=shop&zmi=basequipment.com&cmi=12986189937115136" and xpath="//body"

It is saying:

Redirected to a robots.txt restricted URL

and will not show any results, any way to get around this?

Google [ask bots not to touch that data](http://www.google.com/robots.txt) and Yahoo! is respecting their request. So there shouldn't be a technical way around that. I expect their terms and conditions also forbid you from accessing the data in that way, so you are on shaky legal ground too. — Quentin, Feb 08 '12 at 20:33

score 11 · Answer 1 · answered Mar 15 '12 at 18:26

You can use open tables:

http://developer.yahoo.com/yql/console/?q=show%20tables&env=store://datatables.org/alltableswithkeys#h=select%20content%20from%20data.headers%20where%20url%3D%22www.google.com/products/seller%3Fhl%3Den%26q%3Dburkett+restaurant+equipment+reviews%26tbo%3Du%26tbm%3Dshop%26zmi%3Dbasequipment.com%26cmi%3D12986189937115136%22

YQL "Redirected to a robots.txt restricted URL" Error for Google Domain

1 Answers1