6

I have used robots.txt to restrict one of the folders in my site. The folder consists of the sites in under construction. Google has indexed all those sites which are in testing phase. So I used robots.txt. I first submitted the site and robots.txt is enabled. Now the status is success for the www.mysite.com/robots.txt. But the google is still listing those test links. Here is the code i have written for robots.txt...

User-agent: *
Disallow: /foldername/

Can anyone suggest me what the problem may be? Thanks in advance

Paul Gregoire
  • 9,715
  • 11
  • 67
  • 131
user75472
  • 1,277
  • 4
  • 28
  • 53

2 Answers2

3

Did you test your robots.txt following Googles instructions? http://www.google.com/support/webmasters/bin/answer.py?hl=en&answer=156449

To test a site's robots.txt file:
  • On the Webmaster Tools Home page, click the site you want.
  • Under Site configuration, click Crawler access
  • If it's not already selected, click the Test robots.txt tab.
  • Copy the content of your robots.txt file, and paste it into the first box.
  • In the URLs box, list the site to test against.
  • In the User-agents list, select the user-agents you want.


Webmaster tools: https://www.google.com/webmasters/tools/home?hl=en

Paul Gregoire
  • 9,715
  • 11
  • 67
  • 131
  • Thanks Mondain......... I think robots.txt is downloaded successfully. On Test robots.txt tab, there are three columns like robots.txt file, Downloaded and Status and for each there is a message www.mysite.com/robots.txt, 17 hours ago, 200(success) respectively. I think crawl is done – user75472 Sep 08 '10 at 04:09
  • 1
    Webmaster Tools are part of Google Search Console now. From this support question one can access the robots.txt tester: https://support.google.com/webmasters/answer/6062598?hl=en – Developer Nov 24 '22 at 09:19
3

See Requesting Removal of Content from our Index from the Google Webmaster Blog. You can expedite the removal process by submitting a removal request using the Google Webmaster Tools; otherwise, the pages will eventually be dropped out of the index when it is recrawled (i.e. updating a robots.txt file does not have an immediate change; the change takes place on subsequent crawls).

Michael Aaron Safyan
  • 93,612
  • 16
  • 138
  • 200