Robots.txt in my project root

Question

I've seen tutorials/articles discussing using Robots.txt. Is this still a necessary practice? Do we still need to use this technique?

Robots.txt is for blocking pages from being crawled. I don't know what tutorial you read but it's obviously not a good one. — John Conde, Apr 23 '13 at 18:29
Thanks! I found and am learning now from: [http://www.robotstxt.org/faq.html](http://www.robotstxt.org/faq.html) — id.ot, Apr 23 '13 at 22:25
Stunned that someone down arrow'd this question. Robots.txt (after all of my research and collaboration here in stackoverflow) is without question an "at times" necessary practice. — id.ot, May 18 '13 at 19:47

score 2 · Accepted Answer · answered Apr 24 '13 at 05:56

2

Robots.txt file is not necessary but it is recommended for those who want to block few pages or folders on your website being crawled by search engine crawlers.

answered Apr 24 '13 at 05:56

PeggyP

261
1
4

score 1 · Answer 2 · edited Apr 24 '13 at 09:02

1

I agree with the above answer. Robot.txt file is used for blocking pages and folders from crawling by search engines. For eg. You can block the search engines from crawling and indexing the Session IDs created, which in rare cases could become a security threat! Other than this, I don't see much importance.

edited Apr 24 '13 at 09:02

PeggyP

261
1
4

answered Apr 24 '13 at 06:35

Prasad EBK

11
2

score 1 · Answer 3 · answered Apr 24 '13 at 11:06

The way that a lot of the robots crawl through your site and rank your page has changed recently as well.

I believe for a short period of time the use of Robot.txt may have helped quite a bit, but no adays most other options you'll take in regards to SEO will have more of a positive impact than this little .txt file ever will.

Same goes for backlinks, they used to be far far more important than they are now for you getting ranked.

score 1 · Answer 4 · answered Apr 24 '13 at 15:35

1

Robots.txt is not for indexing . its used to blocks the things that you don't want search engines to index

answered Apr 24 '13 at 15:35

Abhishek Shukla

11
1

Arttu Raittila · Answer 5 · 2013-05-07T12:54:54.430

1

Robots.txt can help with indexation with large sites, if you use it to reveal an XML sitemap file.

Like this:

Sitemap: http://www.domain.com/sitemap.xml

Within the XML file, you can list up to 50,000 URLs for search engines to index. There are plugins for many content management systems that can generate and update these files automatically.

edited May 07 '13 at 12:54

answered Apr 27 '13 at 12:50

Arttu Raittila

176
4

Robots.txt in my project root

5 Answers5