1

We have a website translated to about 15 languages, with more comming in regulary from our editors.

Our URL is built up as follows:

SiteName.Company.com/{LCID}/Folder1/Folder2/

Our company want's to hide a subfolder/page from the search engines.

LCID can be anything from EN-US to ZH-HANS. And i am looking for a solution where we don't have to update the robots file every time a new language is added.

Can i make a wildcard entry or filter out a single foldername regardless of it's path?

IE: companyinfo.nokia.com/en-us/aboutus/darknews/ I want Google to crawl everything but darknews and down (so /darknews/ELOPSellsOut or /darknews/3210revival)

Is this possible?

mfriis
  • 217
  • 3
  • 17
  • Is [tag:lcid] the correct term? It says it’s a "*numeric* ID". – unor Sep 04 '13 at 21:37
  • It doesn't matter in this context but no, your right. LCID is a number. What i meant was the culture info abbrevation. – mfriis Sep 05 '13 at 06:51

1 Answers1

4

You can do this with wildcards:

User-agent: *
Disallow: /*/darknews/

Just be aware that not all robots support wildcards. This will work for Googlebot and Bingbot, but it will not work for many others.

plasticinsect
  • 1,702
  • 1
  • 13
  • 23