How to make raw text file (to edit robots.txt) editor in Django Wagtail admin

Question

The goal is to create a text file editor within the Wagtail admin site. Specifically for robots.txt file. I want the admin to update the file from the admin.

How can I do that?

Answer 1

There are some considerations to make for the robots.txt file hitting your server and doing a database read each time. You probably will want to consider some kind of caching layer here, however I have answered assuming you will resolve that and just want to get a Wagtail 'editor' solution here.

Wagtail provides a contrib.settings module that allows you to have a settings model per site. This is a very helpful module that allows users, usually admin only users, to modify content that is specific to a site but does not suit the idea of a 'page'.

https://docs.wagtail.org/en/stable/reference/contrib/settings.html

Settings model

• In your app's models.py file - set up a new settings model.

from django.db import models

from wagtail.admin.panels import FieldPanel
from wagtail.contrib.settings.models import BaseSetting, register_setting

@register_setting
class MySettings(BaseSetting):
    robots = models.TextField(blank=True)

    panels = [
        FieldPanel('robots'),
    ]

robots.txt view & template

• There are lots of ways to load a View with robots.txt
• Accordbox has a good tutorial on how to set up a robots.txt view https://www.accordbox.com/blog/wagtail-seo-guide/#robotstxt
• Once you have your view, you can get the settings output via the model class.

# using in your view
def robots_view(request):
    robots = MySettings.for_request(request).robots
    ## return the content in your view

• Alternatively you can use in your template context via an injected template context variable
• note: 'wagtail.contrib.settings.context_processors.settings' must be added to context_processors

User-agent: *
{{ settings.app_label.MySettings.robots }}

Considerations

• If possible, it would always be better to serve a static file here, or strongly cache this value at a minimum.
• Validation is a must - make it hard for your editors to break your website, remember that search engines only crawl this semi-regularly and breaking this could mean a few days of de-listing or broken listings if your editors are not careful.
• Permissions are a must, maybe even a two step process to push these changes live somehow?
• You could possibly put a robots field on your HomePage model and access the data in your robots view. This kind of breaks the concept of this model only reflecting the home page but then extending to the 'root' content.