I want to disallow a specific folder and all of its files and subdirectories but I don't know the difference between Disallow: /somedir/ and Disallow: /somedir/*. which one of these lines should I use?
By the way, what does Disallow: /somedir? mean? should I use it too?
Asked
Active
Viewed 60 times
-1
user6931342
- 145
- 3
- 11
-
2This question appears to be off-topic because it is about **SEO** which is off-topic at Stack Overflow. Please read ["Which SEO questions should be closed as non-programming/non-admin?"](//meta.stackoverflow.com/a/382618) to better understand when SEO questions are acceptable to ask here (most are not) and where you might be able to get assistance. – John Conde Dec 05 '20 at 20:52
-
1I don't see this as an SEO question. OP has a question about the robots.txt language specification, OP is not asking about how to get more website hits. Perhaps this issue is already decided as off-topic, but given how it is asked I think it should be allowed. I went ahead and answered it. – mCoding Dec 05 '20 at 21:08
1 Answers
1
According to google (https://developers.google.com/search/reference/robots_txt),
Disallow: /somedir/ and Disallow: /somedir/* are equivalent, the trailing wildcard is ignored. Use whichever you like better.
The ? does not have any special meaning in robots.txt like it does in some pattern matching schemes. Therefore, Disallow: /somedir? means to disallow that literal path including the question mark.
mCoding
- 4,059
- 1
- 5
- 11