0

my hosting company mixed up and trying to limit the search agents they had blocked all including google with robots.txt. Afer I discover it I changed the robots.txt content to Allow: / and waited for a week time Google to see changes but nothing.. Than I completely removed robots.txt file and still I can see this error: enter image description here

The result from this is that my site did 1000-1200 visits per day - now is dropped to 200.. Please help to solve this.. How to wake google that nothing stops him to browse the site? Is it possible all those 5000 url that are now blocked to have been removed from google index?

thecore7
  • 484
  • 2
  • 8
  • 29
  • Try to make a robots.txt that allows everything and see if you'll get a different crawl result – Allan S. Hansen Mar 26 '14 at 08:13
  • I tried this before I removed it - it had no result so that's why I removed it.. – thecore7 Mar 26 '14 at 08:16
  • How long time did you have that new robots.txt file that allowed everything on the server? Should be placed there for some days. Also is there site map submitted? – Allan S. Hansen Mar 26 '14 at 08:19
  • about 3 days and removed it. Now I recreated it again. Yes , I have sitemap submiteet with status OK – thecore7 Mar 26 '14 at 08:20
  • 1
    Wait at least 2 weeks. And check one of blocked URLs with Fetch as Googlebot tools (from GWMT). If it shows that the URL is accessible now, then all is fine. Just be patient and wait. – Oleg Mar 26 '14 at 09:41
  • actually, I can only wait anyway :) Thanks Oleg – thecore7 Mar 26 '14 at 09:55
  • I voted to close this question because it is not a programming question and it is off-topic on Stack Overflow. Non-programming questions about your website should be asked on [webmasters.se]. In the future, please ask questions like this there. – Stephen Ostermiller Mar 30 '22 at 14:56

1 Answers1

0

what you need to do is create robots.txt allowing your whole site to be indexed, upload it to root then go to Webmaster Tools -> Crawl -> Fetch as Google and click on red button saying "FETCH"

wait few seconds or just refresh the page, then click on "Submit to index" and choose "URL and all linked pages"

let me know if that helps, i'm pretty sure it will help

splinterteal
  • 139
  • 9