updating the robots

Question

Ok so I want to add this

User-Agent: *
Disallow: /

to the robots.txt in all the enviroments other then production...any idea on the best want to do this. Should i remove it from the public folder and create a routes/views

I am using rails 3.0.14 prior to asset pipeline...any suggestions

I'd make a capistrano task that uploads robots.txt during deploy. — Sergio Tulentsev, Jul 02 '12 at 23:33
Do you have an example of an after:deploy task ....would it be best to create the file on the fly or move the file ...suggestions — Matt Elhotiby, Jul 02 '12 at 23:36

score 4 · Answer 1 · answered Nov 15 '12 at 00:32

Capistrano task for uploading a blocking robots.txt

I wrote this up again today, same path as Sergio's answer essentially, but sharing robots-specific result might save someone time :)

namespace :deploy do
  desc "Uploads a robots.txt that mandates the site as off-limits to crawlers"
  task :block_robots, :roles => :app do
    content = [
      '# This is a staging site. Do not index.',
      'User-agent: *',
      'Disallow: /'
    ].join($/)

    logger.info "Uploading blocking robots.txt"
    put content, "#{current_path}/public/robots.txt"
  end
end

Then trigger it from your staging recipe with something like

after "deploy:update_code", "deploy:block_robots"

This was really helpful. I had to change current_path to release_path for some reason, though. — hamishtaplin, Nov 15 '13 at 10:45

score 3 · Accepted Answer · answered Jul 02 '12 at 23:45

Here's a real working code from my project (it's a nginx config, not robots.txt, but idea should be clear).

task :nginx_config do
  conf = <<-CONF
   server {
      listen 80;
      client_max_body_size 2M; 
      server_name #{domain_name};

      -- snip --
    }
  CONF

  put conf, "/etc/nginx/sites-available/#{application}_#{rails_env}"
end

So, basically, you create content of your file in a string and then do put to desired path. This will make capistrano upload the content through SFTP.

updating the robots

2 Answers2

Capistrano task for uploading a blocking robots.txt