Filtering a Wagtail index page by fields of the child pages (user-initiated query) / (FieldError at ... cannot resolve keyword)

Question

My Wagtail project is at heart just a very conventional listings page where users can browse items in the database and then click on any item of interest to go its detail page. But how do I allow users to filter and/or sort the listings on the main page by the contents of fields on the child pages? This most generic, ordinary task eludes me.

Let's say the database is a collection of Things. And let's say that what people find important about each Thing are (a) the year it was discovered, and (b) the country where it can be found. A user may want to browse through all the Things, but she should be able to narrow down the list to just those Things found in 2019 in Lithuania. And she should be able to sort by year or by country. Just your super-standard functionality but I can't find any guidance or figure it out myself.

Cribbing from examples of other people's work here are my models so far:

    class ThingsListingPage(Page):
    
        def things(self):
            ''' return all child pages, to start with '''
            things = self.get_children().specific() 
               # do I need 'specific' above? 
               # Is this altogether the wrong way to fetch child 
               #    pages if I need to filter on their fields?
            return things
    
        def years(self, things): # Don't need things parameter yet
            '''Return a list of years for use in queries'''
            years = ['2020', '2019', '2018',]
            return years
    
        def countries(self, things):
            '''Return a list of countries for use in queries.'''
            countries = ['Angola', 'Brazil', 'Cameroon','Dubai', 'Estonia',]
            return countries
    
        def get_context(self, request):
            context = super(ThingsListingPage, self).get_context(request)
            things = self.things()
    
            # this default sort is all you get for now
            things_to_display = things.order_by('-first_published_at')
    
            # Filters prn
            has_filter = False 
            for filter_name in ['year', 'country',]:
                filter_value = request.GET.get(filter_name)
                if filter_value:
                    if filter_value != "all":
                        kwargs = {'{0}'.format(filter_name): filter_value}
                        things_to_display = things_to_display.filter(
                            **kwargs)
                    has_filter = True
    
            page = request.GET.get('page') # is this for pagination?
    
            context['some_things'] = things_to_display
            context['has_filter'] = has_filter # tested on listings page to select header string
            context['page_check'] = page # pagination thing, I guess
            # Don't forget the data to populate filter choices on the listings page
            context['years'] = self.years(things)
            context['countries'] = self.countries(things)
    
            return context
    
    class ThingDetailPage(Page):
    
            year = models.CharField(
            max_length=4,
            blank=True,
            null=True,
        )
    
        country = models.CharField(
            max_length=50,
            blank=True,
            null=True,
        )
        CONTNT_PANELS = Page.content_panels + [
            FieldPanel('year'),
            FieldPanel('country'),
        ]
        # etc. 

The template for the listings (index) page, showing only the filter controls (sorting controls are also required, and of course the listings themselves):

{% extends "base.html" %}
{% block content %}
  <section class="filter controls">
    <form method="get" accept-charset="utf-8" class="filter_form">
      <ul>
        <li>
          <label>Years</label>
          <h6 class="expanded">Sub-categories</h6>
          <ul class="subfilter">
              <li>
                  <input type="radio" name="year" value="all" id="filter_year_all"
                      {% if request.GET.year == "all" %}checked="checked" {% endif %} /><label
                      for="filter_year_all">All Years</label></input>
              </li>
              {% for year in years %}
              <li>
              <input type="radio" name="year" value="{{ year }}" id="filter_year_{{ year }}"
                {% if request.GET.year == year %}checked="checked" {% endif %} /><label
                for="filter_year_{{ year }}">{{ year }}</label></input>
              </li>
              {% endfor %}
            </ul>
        </li>
        <li>
          <label>Countries</label>
          <h6 class="expanded">Sub-categories</h6>
          <ul class="subfilter">
            <li>
              <input type="radio" name="country" value="all" id="filter_country_all"
                  {% if request.GET.country == "all" %}checked="checked" {% endif %} /><label
                  for="filter_country_all">All Countries</label></input>
            </li>
            {% for country in countries %}
            <li>
                <input type="radio" name="country" value="{{ country }}" id="filter_country_{{ country|slugify }}"
                    {% if request.GET.country == country %}checked="checked" {% endif %} /><label
                    for="filter_country_{{ country|slugify }}">{{ country }}</label></input>
            </li>
            {% endfor %}
          </ul>
        </li>
      </ul>
      <input type="submit" value="Apply Filters"/>
    </form>
  </section>
{% endblock %}

The above Page models seem to work fine in Wagtail. I've created a ThingsListingPage page named "Things," and a set of child ThingDetailPage pages, each with 'year' and 'country' data. The pages display fine: The filters on the Things listings page display the (currently hard-coded) year and country items from the ThingsListingPage model. The listings page also lists the child pages on command. No complaints from the server.

But: Upon making my filter selections and clicking the submit / Apply filters button, I get an appropriate URL in the address bar (http://localhost:8000/things/?year=2019&country=Lithuania) but this error: FieldError at /things/ Cannot resolve keyword 'year' into field. (If I don't select a year filter but do filter on a country I get the same error on the 'country' keyword.)

---

SO: How should I change the ThingsListingPage model so that I can filter on child page fields (fields of ThingDetailPage pages)? Or is there a completely different approach I should be taking, a better / everybody-knows-that's-how Wagtail way to do arbitrary, user-initiated filter and sort operations on a page's children's fields?

Just please note that in the real project there may be different page types for TinyThings, WildThings, and what not, so I'm looking for a solution that can be modified to work even when some children don't have the field(s) used in the filter(s).

I'd also appreciate any direction you might have on how sort operations should be done.

Answer 1

Page.get_children returns the results as basic Page instances where only the core fields such as title are available - this is because it has no way to know the expected page type of the children at the time of doing the query (see What is the difference between ChildPage.objects.child_of(self) and ParentPage.get_children()?). Adding .specific() will return the results as the more specific page types, but this won't help for filtering, since it works as a post-processing step after the main query has run (including applying any filters).

However, you can get around this by reorganising the query expression to specify the page type:

things = ThingDetailPage.objects.child_of(self)

Here, Django knows which specific page model to query on, and so all fields of ThingDetailPage are available for filtering.

This does limit you to a single page type, and there's no perfect way around that - at the database level each page type is handled by a separate table, and it's not possible to efficiently query data that's distributed over multiple tables. (Even if ThingDetailPage and TinyThingDetailPage both have a year field defined, those are distinct entities in the database, so there's not a single 'year' column that can be filtered on.) However, you may be able to restructure your models to accommodate this using multi-table inheritance. Just as Wagtail itself gives you a base Page model containing the fields common to all pages, you could define ThingDetailPage to contain the fields common to all Things (such as year) and have subtypes inheriting from that:

class TinyThingDetailPage(ThingDetailPage):
    size = models.CharField(...)

Your TinyThingDetailPages will then be included in the results of ThingDetailPage.objects.child_of(self), although they'll only be returned as instances of ThingDetailPage. Again, you can add .specific() to return them in their most specific form, but this won't work for filtering - so you'll be able to filter on the fields common to ThingDetailPage (such as year) but not size.

Answer 2

The answer by @gasman succeeds perfectly. However, an alternative solution suggested by one of @gasman's linked posts is equally effective.

The @gasman answer above requires that child page objects be accessed through their own model (note that you need to append child_of(self) so that the query returns only those pages that belong to the current listings page:

    def things(self):
        ''' return all child pages, to start with '''
        things = ThingDetailPage.objects.child_of(self)
        return things

By querying the child page model in this way you can then access the fields of the child pages without any prefix. Thus, the get_context method in my models.py (see question) works as is. Specifically, these lines work without any changes:

     kwargs = {'{0}'.format(filter_name): filter_value}
     things_to_display = things_to_display.filter(**kwargs)

However, it seems it's just as valid to start with the listing page and retrieve its child pages, the way the query was written in my question:

    def things(self):
        ''' return all child pages, to start with '''
        things = self.get_children().specific()
        return things

But to make this work, you must then prefix your filters with the name of the child page model (with an obligatory two underscores connecting the model name with that of the field).
The filters functioned as expected and without error as soon as I added that:

    kwargs = {'{0}'.format('thingdetailpage__' + filter_name): filter_value}
    things_to_display = things_to_display.filter(**kwargs)

Based on these findings, I have some doubt about the explanation @gasman gives as to why my original code didn't work:

Adding .specific() will return the results as the more specific page types, but this won't help for filtering, since it works as a post-processing step after the main query has run (including applying any filters).

Be that as it may, I'm adding this answer just to document an alternative solution, i.e., one that also seems to work. I have no reason to prefer the approach used in my question. Filters on child page fields now function.