Django get_next_by_FIELD using complex Q lookups

Question

While creating a front end for a Django module I faced the following problem inside Django core:

In order to display a link to the next/previous object from a model query, we can use the extra-instance-methods of a model instance: get_next_by_FIELD() or get_previous_by_FIELD(). Where FIELD is a model field of type DateField or DateTimeField.

Lets explain it with an example

from django.db import models

class Shoe(models.Model):
    created = models.DateTimeField(auto_now_add=True, null=False)
    size = models.IntegerField()

A view to display a list of shoes, excluding those where size equals 4:

def list_shoes(request):
    shoes = Shoe.objects.exclude(size=4)

    return render_to_response(request, {
        'shoes': shoes
    })

And let the following be a view to display one shoe and the corresponding link to the previous and next shoe.

def show_shoe(request, shoe_id):
    shoe = Shoe.objects.get(pk=shoe_id)

    prev_shoe = shoe.get_previous_by_created()
    next_shoe = shoe.get_next_by_created()

    return render_to_response('show_shoe.html', {
        'shoe': shoe,
        'prev_shoe': prev_shoe,
        'next_shoe': next_shoe
    })

Now I have the situation that the show_shoe view displays the link to the previous/next regardless of the shoes size. But I actually wanted just shoes whose size is not 4. Therefore I tried to use the **kwargs argument of the get_(previous|next)_by_created() methods to filter out the unwanted shoes, as stated by the documentation:

Both of these methods will perform their queries using the default manager for the model. If you need to emulate filtering used by a custom manager, or want to perform one-off custom filtering, both methods also accept optional keyword arguments, which should be in the format described in Field lookups.

Edit: Keep an eye on the word "should", because then also (size_ne=4) should work, but it doesn't.

The actual problem

Filtering using the lookup size__ne ...

def show_shoe(request, shoe_id):
    ...
    prev_shoe = shoe.get_previous_by_created(size__ne=4)
    next_shoe = shoe.get_next_by_created(size__ne=4)
    ...

... didn't work, it throws FieldError: Cannot resolve keyword 'size_ne' into field.

Then I tried to use a negated complex lookup using Q objects:

from django.db.models import Q

def show_shoe(request, shoe_id):
    ...
    prev_shoe = shoe.get_previous_by_created(~Q(size=4))
    next_shoe = shoe.get_next_by_created(~Q(size=4))
    ...

... didn't work either, throws TypeError: _get_next_or_previous_by_FIELD() got multiple values for argument 'field'

Because the get_(previous|next)_by_created methods only accept **kwargs.

The actual solution

Since these instance methods use the _get_next_or_previous_by_FIELD(self, field, is_next, **kwargs) I changed it to accept positional arguments using *args and passed them to the filter, like the **kwargs.

def my_get_next_or_previous_by_FIELD(self, field, is_next, *args, **kwargs):
    """
    Workaround to call get_next_or_previous_by_FIELD by using complext lookup queries using
    Djangos Q Class. The only difference between this version and original version is that
    positional arguments are also passed to the filter function.
    """
    if not self.pk:
        raise ValueError("get_next/get_previous cannot be used on unsaved objects.")
    op = 'gt' if is_next else 'lt'
    order = '' if is_next else '-'
    param = force_text(getattr(self, field.attname))
    q = Q(**{'%s__%s' % (field.name, op): param})
    q = q | Q(**{field.name: param, 'pk__%s' % op: self.pk})
    qs = self.__class__._default_manager.using(self._state.db).filter(*args, **kwargs).filter(q).order_by('%s%s' % (order, field.name), '%spk' % order)
    try:
        return qs[0]
    except IndexError:
        raise self.DoesNotExist("%s matching query does not exist." % self.__class__._meta.object_name)

And calling it like:

...
prev_shoe = shoe.my_get_next_or_previous_by_FIELD(Shoe._meta.get_field('created'), False, ~Q(state=4))
next_shoe = shoe.my_get_next_or_previous_by_FIELD(Shoe._meta.get_field('created'), True, ~Q(state=4))
...

finally did it.

Now the question to you

Is there an easier way to handle this? Should shoe.get_previous_by_created(size__ne=4) work as expected or should I report this issue to the Django guys, in the hope they'll accept my _get_next_or_previous_by_FIELD() fix?

Environment: Django 1.7, haven't tested it on 1.9 yet, but the code for _get_next_or_previous_by_FIELD() stayed the same.

Edit: It is true that complex lookups using Q object is not part of "field lookups", it's more part of the filter() and exclude() functions instead. And I am probably wrong when I suppose that get_next_by_FIELD should accept Q objects too. But since the changes involved are minimal and the advantage to use Q object is high, I think these changes should get upstream.

tags: django, complex-lookup, query, get_next_by_FIELD, get_previous_by_FIELD

(listing tags here, because I don't have enough reputations.)

Since this is my first question, any hints to improve it? – Florin Hillebrand Feb 24 '16 at 13:22 — Florin Hillebrand, Feb 24 '16 at 13:22

score 1 · Answer 1 · answered Feb 24 '16 at 18:29

1

You can create custom lookup ne and use it:

.get_next_by_created(size__ne=4)

answered Feb 24 '16 at 18:29

vsd

1,473
13
11

Creating an own lookup works! But why isn't it possible to use size__ne out of the box? Since it 'should' behave like other field lookups. – Florin Hillebrand Feb 25 '16 at 13:49
2

Because there is no `ne` lookup by default. – vsd Feb 25 '16 at 13:58

Ambroise · Answer 2 · 2016-02-24T15:39:00.230

0

I suspect the method you've tried first only takes lookup arg for the field you're basing the get_next on. Meaning you won't be able to access the size field from the get_next_by_created() method, for example.

Edit : your method is by far more efficient, but to answer your question on the Django issue, I think everything is working the way it is supposed to. You could offer an additional method such as yours but the existing get_next_by_FIELD is working as described in the docs.

You've managed to work around this with a working method, which is OK I guess, but if you wanted to reduce the overhead, you could try a simple loop :

def get_next_by_field_filtered(obj, field=None, **kwargs):

    next_obj = getattr(obj, 'get_next_by_{}'.format(field))()

    for key in kwargs:
        if not getattr(next_obj, str(key)) == kwargs[str(key)]:
            return get_next_by_field_filtered(next_obj, field=field, **kwargs)

    return next_obj

This isn't very efficient but it's one way to do what you want.

Hope this helps !

Regards,

edited Feb 24 '16 at 15:39

answered Feb 24 '16 at 14:18

Ambroise

1,649
1
13
16

Yes, looping over would work too, but yeah I don't really like it, but your solution looks better. – Florin Hillebrand Feb 25 '16 at 13:14
And I'm not really sure if the computation/db overhead by looping over is lower than using the modified *my_get_next_or_previous_by_FIELD*, because every call to get_next_by_FIELD will require an access to the DB. This is why I wanted to solve this using size_ne or Q objects, so that just one DB access delivers the desired result. – Florin Hillebrand Feb 25 '16 at 13:39
Yes you're absolutely right, your method works really well :) – Ambroise Feb 25 '16 at 16:25
Django needs to add an easier way to do this, than using only the datetime field, honestly I can't recall the amount of time I needed to get next and previous objects and finds myself hacking with session (yes I call working with session hacking) I think including `get_next_by_ID or get_previous_by_ID` would have been a better options, unless it's already there and I missed it. – Cynthia Onyilimba May 19 '22 at 10:03

Django get_next_by_FIELD using complex Q lookups

Lets explain it with an example

The actual problem

The actual solution

Now the question to you

2 Answers2