UrlMapping to static files in Grails

Question

I want to map the static files sitemap.xml and robots.txt which a located in my web-app directory. The urls should be as follows:

http://www.mydomain.com/sitemap.xml 
http://www.mydomain.com/robots.txt

How do I have to set the url mapping to make these routes work?

score 11 · Answer 1 · answered Jul 31 '13 at 21:49

11

The simplest way is to tell grails to ignore them in UrlMappings.groovy:

class UrlMappings {
    static excludes = ['/robots.txt', '/sitemap.xml']

    static mappings = {
        // normal mappings here ...
    }
}

answered Jul 31 '13 at 21:49

codelark

12,254
1
45
49

This answer more accurately represents what the OP requested, although the selected answer provides a working alternate approach. – mnd Nov 04 '15 at 16:44

doelleri · Accepted Answer · 2013-07-31T16:21:35.173

8

I use this mapping for robots.txt:

"/robots.txt" (view: "/robots")

And then have a grails-app/views/robots.gsp that contains the content for robots.txt. This way I can use <g:if env="..."> to easily have different content for different environments.

In order for this to work for a ".xml" extension, you need to change the Content Negotiation config.

grails.mime.file.extensions = false // disables the parsing of file extensions from URLs into the request format

edited Jul 31 '13 at 16:21

answered Jul 31 '13 at 14:47

doelleri

19,232
5
61
65

Thanks. It works for robots, but not for sitemap.xml how would you do that? – Michael Jul 31 '13 at 14:55
Does this solution has any any side effects? – Michael Jul 31 '13 at 18:10
Automatic content negotiation based on file extensions won't happen – doelleri Jul 31 '13 at 19:06

Tyler Rafferty · Answer 3 · 2019-05-03T01:30:18.840

It might also be helpful to to set up a nofollow to your Staging environment if you are using one. Not sure if there is a use case to having a staging site indexed.... so if you agree you may be able to use these steps to help block that.

If you are using Tomcat, set an environment variable such as NOFOLLOW=true --> see here for example: TOMCAT_OPTS, environment variable and System.getEnv()

Next as mentioned by @doelleri set the urlMappings

UrlMappings

//Robots.txt
"/robots.txt"(controller: 'robots', action:'robots')

Then use your robotsController to detect the environment variable you set on your staging tomcat.

RobotsController

def robots() {
    if (System.getenv('NOFOLLOW') == 'true') {
        def text = "User-agent: *\n" +
            "Disallow: /cgi-bin/ \n" +
            "Disallow: /tmp/ \n" +
            "Disallow: /junk/ \n" +
            "Disallow: /admin/ \n" +
            "Crawl-delay: 5 \n" +
            "Sitemap: https://www.example.com/sitemap.xml"

        render(text: text, contentType: "text/plain", encoding: "UTF-8")
    } else {
        render(status: 404, text: 'Failed to load robots.txt')
    }
}

robots.gsp

<%-- Content rendered from controller -> so leave blank :) --%>

UrlMapping to static files in Grails

3 Answers3