Stop google from following my api

Question

OK, I have a website which has a series of buttons which call an api to return informtion relating to products. The problem i have is that google is accessing and following the api every 10 minutes approximatley which means i am doing loads of lookups a day.

Can i stop google bots from doing this??

I have a robots.txt file and no follow meta but they do not seem to be doing much.

Any advice would be appreciated.

   User-Agent: *
  Disallow:/index/
  Disallow /manufacturer/

I just noticed the pages didn't have extensions i.e manufacturer should be manufacturer.php

Thanks

Did you see this one? http://stackoverflow.com/questions/344697/googlebots-ignoring-robots-txt?rq=1 — Mike Cheel, Nov 21 '12 at 18:42
Also, kindly visit: http://support.google.com/webmasters/bin/answer.py?hl=en&answer=156449&from=35237&rd=1 — Mike Cheel, Nov 21 '12 at 18:43
thanks that's pretty much what my robot.txt says only with the disallow — Sam Williams, Nov 21 '12 at 18:45
Thanks Mike, i have read that, in fact thats where i wrote my robots.txt file from, but it hasn't stopped google from visiting — Sam Williams, Nov 21 '12 at 18:47

score 0 · Answer 1 · answered Nov 21 '12 at 18:52

0

Google Webmaster tools will analyze your robots.txt for you and let you know what it is making out of it. I have found it very useful in the past. You can just paste your robots.txt and it will point out any flaws.

Failing that you can visit the forums.

answered Nov 21 '12 at 18:52

alexfernandez

1,938
3
19
30

Stop google from following my api

1 Answers1