Can I keep robots.txt in a contextpath and give a 301 redirect?

Question

My website uses a contextpath (eg: www.example.com/abc). The robots.txt is available at www.example.com/abc/robots.txt and I have given a 301 redirect in webserver to redirect www.example.com/robots.txt to www.example.com/abc/robots.txt.

My question is whether the search engines be able to read the robots.txt file since it has a 301 redirect?

score 1 · Answer 1 · answered Dec 12 '17 at 22:00

1

Found that the search engines are honoring the 301 redirect and reading the file from the subfolder.

answered Dec 12 '17 at 22:00

Aromal

176
2
8

I observe the same. It seems Google Search Console tool https://www.google.com/webmasters/tools/robots-testing-tool does not complain. – pixeline Jul 27 '22 at 09:29

score 0 · Answer 2 · answered Jun 12 '17 at 10:08

Robots.txt should be on root level

https://example.com/robots.txt - Correct
https://blog.example.com/robots.txt - Correct
https://example.com/abc/robots.txt - Not Correct
https://blog.example.com/abc/robots.txt - Not Correct

If it is on sub directory/sub folder then it will return 404 error(Because they make calls only on root directory), and Google will ignore your robots.txt completely if it is return 301 or 404 error.

Can I keep robots.txt in a contextpath and give a 301 redirect?

2 Answers2