0

I am creating an ASP.NET website witch forces users to accept a disclaimer. When they accept the disclaimer a cookie and session is set. On each page request a check is fired to see if the session or cookie present.

We want to allowed Google and other Searchbots to index/craw all the pages without accepting the disclaimer.

What is to best way to do this? The only thing that I can think about is a check in the Request.ServerVariables, but I am not sure witch values I should look for?

Ivo
  • 3,406
  • 4
  • 33
  • 56

2 Answers2

1

You need to bypass the check for the session/coockie in case Googlebot is passing by.

So in this check you indeed need to look at ServerVariables. E.g. for Google this would be something like

   HttpContext.Current.Request.ServerVariables["HTTP_USER_AGENT"].Contains("Googlebot"))

Of course, anyone pretending to be google will now also be able to skip the disclaimer.

I wouldn't call this cloacking - but no guarantees as to wether google likes it or not

Pleun
  • 8,856
  • 2
  • 30
  • 50
-1

you must add meta tags for that, else deny access using web.config meta tags work this way

<meta name="robots" content="noindex nofollow">

specially for google it is 

<meta name="googlebot" content="noindex">

also you can try the URL Removal Request service

Deeptechtons
  • 10,945
  • 27
  • 96
  • 178
  • this won't work because the entire page will not be visible to the search engines – Pleun Jun 15 '11 at 09:56
  • @who ever down voted Disclaimer is a separate page right? or a Popup? Without more info how could someone "give working anywhere" solutions – Deeptechtons Jun 15 '11 at 10:07
  • Its handled server side, before the page is rendered. The question is not exclude the disclaimer page it selve (btw i didnt vote you down) – Ivo Jun 15 '11 at 10:37
  • @lvo `it's handled in server` and `ASP.NET website witch forces users to accept a disclaimer. When they accept the disclaimer a cookie and session is set` both the explanation seems to contradict. How the user accepts to disclaimer if it isn't shown to them – Deeptechtons Jun 15 '11 at 10:50
  • the user accepts it, but the google bot should auto accept it/ not be denied access to a page – Ivo Jun 15 '11 at 12:33