How to add route to dynamic robots.txt in ASP.NET MVC?

Question

I have a robots.txt that is not static but generated dynamically. My problem is creating a route from root/robots.txt to my controller action.

This works:

routes.MapRoute(
name: "Robots",
url: "robots",
defaults: new { controller = "Home", action = "Robots" });

This doesn't work:

routes.MapRoute(
name: "Robots",
 url: "robots.txt", /* this is the only thing I've changed */
defaults: new { controller = "Home", action = "Robots" });

The ".txt" causes ASP to barf apparently

Answer 1

You need to add the following to your web.config file to allow the route with a file extension to execute.

<?xml version="1.0" encoding="utf-8"?>
<configuration>
  <!-- ...Omitted -->
  <system.webServer>
    <!-- ...Omitted -->
    <handlers>
      <!-- ...Omitted -->
      <add name="RobotsText" 
           path="robots.txt" 
           verb="GET" 
           type="System.Web.Handlers.TransferRequestHandler" 
           preCondition="integratedMode,runtimeVersionv4.0" />
    </handlers>
  </system.webServer>
</configuration>

See my blog post on Dynamically Generating Robots.txt Using ASP.NET MVC for more details.

Answer 2

Answer here: url with extension not getting handled by routing. Basically, when asp sees the "." it calls the static file handler, so the dynamic route is never used. The web.config files needs to be modified so /robots.txt will not be intercepted by the static file handler.

Answer 3

Muhammad Rehan Saeed's System.Web.Handlers.TransferRequestHandler approach in web.config approach did not work for me due to the environment I was working in, resulting in 500 errors.

An alternative of using a web.config url rewrite rule worked for me instead:

<rewrite>
    <rules>
        <rule name="Dynamic robots.txt" stopProcessing="true">
            <match url="robots.txt" />
            <action type="Rewrite" url="/DynamicFiles/RobotsTxt" />
        </rule>
    </rules>
</rewrite>