Error in robots.txt errors keep piling up even though it is fixed

Question

Somebody messed up our robots.txt by accidentally adding \n after our entire allow: /products/ which are about 30.000 pages in total. The errors are on multiple language sites. This is one of our Search consoles.

I quickly noticed the error and deleted it. I've asked Google to verify my solution but about 3 months later the errors are still increasing. See the images below:

Is there anything I can do to speed up the proces? I have started the validation.

post an url of your robots.txt – Evgeniy Jan 17 '19 at 09:33 — Evgeniy, Jan 17 '19 at 09:33
https://www.unisgroup.nl/robots.txt/ – Kevin Tad Jan 17 '19 at 09:50 — Kevin Tad, Jan 17 '19 at 09:50

score 1 · Answer 1 · answered Jan 17 '19 at 10:39

1

Your robots.txt is unreachable. It is redirected by 301 to https://www.unisgroup.nl/robots.txt/, which is directory, not a file - do you see the trailing slash? Google is looking for a file, doesn't find it and goes away.

answered Jan 17 '19 at 10:39

Evgeniy

2,337
2
28
68

Thanks for your response! So I should change the location of the robots.txt or should I change something in search console? – Kevin Tad Jan 17 '19 at 11:19
You should make robots.txt file reachable as file with the status code 200 – Evgeniy Jan 17 '19 at 11:20

Error in robots.txt errors keep piling up even though it is fixed

1 Answers1