7

I have a site that I am using a Google Custom Search Engine on. I want Google CSE to crawl my site but I want it to stay out of the results of a regular Google search. I put this in my robots.txt file hoping that google CSE bots would ignore it since I specified the pages I wanted Google CSE to crawl in the settings 

User-agent: *
Disallow: /

I guess the Google CSE bots also have to obey robots.txt. So is there a way to get my page to stay out of search engine searches but for Google CSE to still be able to index it? TIA!

Bender
  • 361
  • 2
  • 3
  • 13

1 Answers1

1

There is no solution to this question that fits with what you would like. I too have the same situation where I need custom search only. Unfortunately Google's list of crawlers doesn't show a specific bot for google custom search. Blocking Googlebot will both kill native search and custom search.

Giles Wells
  • 128
  • 5