0

I have the following url structure on my website:

user accounts:

http://www.mydomain.com/username

user my contain items that are under: 

http://www.mydomain.com/username/item/itemId

What do I have to set in my robots.txt that the user accounts will not be indexed and all the items will be indexed?

Michael
  • 32,527
  • 49
  • 210
  • 370

1 Answers1

1

Actually, I don't think that's going to be possible with robots.txt. Google's wildcard support probably won't support that if username, item, and item-id can be any arbitrary strings. You'd have to specify each username, at minimum.

Check out Google's Robots.txt Specifications page. There might be some other method of doing what you want.

Jim Mischel
  • 131,090
  • 20
  • 188
  • 351