Ruby on Rails robots.txt folders

Question

I'm about to launch a Ruby on Rails application and as the last task, I want to set the robots.txt file. I couldn't find information about how the paths should be written properly for a Rails application.

Is the starting path always the root path from the Ruby on Rails application or the app folder? How would I then disallow, e.g., the img folder?

Do I have to write the paths as I see them in the app folder, or like how the paths look on the site online, e.g. http://example.com/admin?

Answer 1

You have to put your robots.txt file in the /public folder.

It may look like this example from robotstxt.org:

The following example "/robots.txt" file specifies that no robots should visit any URL starting with "/cyberworld/map/" or "/tmp/", or /foo.html:

# robots.txt for http://www.example.com/

User-agent: *
Disallow: /cyberworld/map/ # This is an infinite virtual URL space
Disallow: /tmp/ # these will soon disappear
Disallow: /foo.html

Further reading:

• How Google interprets the robots.txt specification
• About /robots.txt

Answer 2

Ruby on Rails automatically creates a public/robots.txt by default since at least Ruby on Rails 3 or even earlier. All files in public/* will be delivered by Puma/Thin/Webrick/Passenger by default, so it is just enough to put the file there.

Inside you can just disallow specific paths by writing the absolute path beginning with "/":

User-agent: *
Disallow /img

See the Guides for more information on robots.txt.