What is the smartest way to handle robots.txt in Express?

Question

I'm currently working on an application built with Express (Node.js) and I want to know what is the smartest way to handle different robots.txt for different environments (development, production).

This is what I have right now but I'm not convinced by the solution, I think it is dirty:

app.get '/robots.txt', (req, res) ->
  res.set 'Content-Type', 'text/plain'
  if app.settings.env == 'production'
    res.send 'User-agent: *\nDisallow: /signin\nDisallow: /signup\nDisallow: /signout\nSitemap: /sitemap.xml'
  else
    res.send 'User-agent: *\nDisallow: /'

(NB: it is CoffeeScript)

There should be a better way. How would you do it?

Thank you.

Answer 1

Use a middleware function. This way the robots.txt will be handled before any session, cookieParser, etc:

app.use('/robots.txt', function (req, res, next) {
    res.type('text/plain')
    res.send("User-agent: *\nDisallow: /");
});

With express 4 app.get now gets handled in the order it appears so you can just use that:

app.get('/robots.txt', function (req, res) {
    res.type('text/plain');
    res.send("User-agent: *\nDisallow: /");
});

Answer 2

1. Create robots.txt with following content :

User-agent: *
Disallow: # your rules here

2. Add it to public/ directory.

3. If not already present in your code, add:

app.use(express.static('public'))

Your robots.txt will be available to any crawler at http://yoursite.com/robots.txt

Answer 3

Looks like an ok way.

An alternative, if you'd like to be able to edit robots.txt as regular file, and possibly have other files you only want in production or development mode would be to use 2 separate directories, and activate one or the other at startup.

if (app.settings.env === 'production') {
  app.use(express['static'](__dirname + '/production'));
} else {
  app.use(express['static'](__dirname + '/development'));
}

then you add 2 directories with each version of robots.txt.

PROJECT DIR
    development
        robots.txt  <-- dev version
    production
        robots.txt  <-- more permissive prod version

And you can keep adding more files in either directory and keep your code simpler.

(sorry, this is javascript, not coffeescript)

Answer 4

Here is what I use

router.use('/robots.txt', function (req, res, next) {
  res.type('text/plain')
  res.send(
    `User-agent: *
     Disallow: /admin`);
});

Answer 5

For choosing the robots.txt depending the environment with a middleware way:

var env = process.env.NODE_ENV || 'development';

if (env === 'development' || env === 'qa') {
  app.use(function (req, res, next) {
    if ('/robots.txt' === req.url) {
      res.type('text/plain');
      res.send('User-agent: *\nDisallow: /');
    } else {
      next();
    }
  });
}

Answer 6

This is what I did on my index routes. You can just simply write down in your codes what I does given down below.

router.get('/', (req, res) =>
    res.sendFile(__dirname + '/public/sitemap.xml')
)

router.get('/', (req, res) => {
    res.sendFile(__dirname + '/public/robots.txt')
})

Answer 7

I use robots.txt as a normal file for Prod, and a middleware for other envs.

if(isDev || isStaging){
    app.use('/robots.txt', function (req, res) {
        res.type('text/plain');
        res.send("User-agent: *\nDisallow: /");
    });
}
app.use(express.static(path.join(__dirname, 'public')));

Answer 8

Focusing more on the most convenient and simple solution instead of the "best" or "smartest". I simply added the following to the server.ts file.

server.get('/robots.txt', function (req, res) {
  res.type('text/plain');
  res.send("User-agent: *\nAllow: /");
})

What this does is create a robots.txt file on the fly and sends it whenever the /robots.txt file is called for. Now to get this to work, the code fragment must be placed before the other server.get function calls (so it takes priority). I'm implementing Express with Angular, for which the full code fragment for me ended up being:

export function app(): express.Express {
  const server = express();
  const distFolder = join(process.cwd(), 'dist/sophisticatedPrimate/browser');
  const indexHtml = existsSync(join(distFolder, 'index.original.html')) ? 'index.original.html' : 'index';

  // Our Universal express-engine (found @ https://github.com/angular/universal/tree/main/modules/express-engine)
  server.engine('html', ngExpressEngine({
    bootstrap: AppServerModule,
  }));

  server.set('view engine', 'html');
  server.set('views', distFolder);

  server.get('/robots.txt', function (req, res) {
    res.type('text/plain');
    res.send("User-agent: *\nAllow: /");
  })

  // Example Express Rest API endpoints
  // server.get('/api/**', (req, res) => { });
  // Serve static files from /browser
  server.get('*.*', express.static(distFolder, {
    maxAge: '1y'
  }));

  // All regular routes use the Universal engine
  server.get('*', (req, res) => {
    res.render(indexHtml, { req, providers: [{ provide: APP_BASE_HREF, useValue: req.baseUrl }] });
  });

  return server;
}

Answer 9

app.use(express.static('public'))
app.use('/images', express.static('public/images'))
app.use('/videos', express.static('public/videos'))