Django custom manager`s method that changes data works incorrectly with other manager`s methods (filter, get, etc.)

Question

I have created a custom Manager, but in my case its method adds some string to the end of particular field in a model instead of just filtering queryset as in usual cases.

My goal is to return already changed objects when calling SomeModel.objects. Django`s documentation says:

You can override a Manager’s base QuerySet by overriding the Manager.get_queryset() method. get_queryset() should return a QuerySet with the properties you require.

My approach works, when I call SomeModel.objects.all(), but if I apply some filter for objects or just after .all() I can see that data become just regular.

models.py:

class BaseModelQuerySet(models.QuerySet):
    def edit_desc(self, string):
        if self.exists():
            for obj in self:
                if 'description' in obj.__dict__:
                    obj.__dict__['description'] += string
        return self

class BaseModelManager(models.Manager):
    def get_queryset(self):
        return BaseModelQuerySet(self.model, using=self._db).edit_desc('...extra text')


class BaseModel(models.Model):
   objects = BaseModelManager()

    class Meta:
        abstract = True

Shell output:

>>> Macro.objects.all()[0].description
'Test text...extra text'
>>> Macro.objects.all().filter(id=1)[0].description
'Test text'

That makes me confused. Such impression that other methods calling regular queryset instead of one returned with custom objects.

Does this behavior still apply when you omit `all` in filter? eg `Macro.objects.filter(id=1).first().description` — Jason, Dec 21 '17 at 13:30
@Jason no need to ask, if the OP removes the (indeed useless in this case) `.all()` call things will seem to work as the OP expects - but it will break again as soon as anyone tries to chain another filter/exlude etc call, cf my answer. — bruno desthuilliers, Dec 21 '17 at 13:35

bruno desthuilliers · Answer 1 · 2017-12-21T14:42:30.133

3

Queryset methods that return a queryset actually return a new queryset (they dont change the current queryset inplace). It's not that they are returning "regular" querysets, they are instances of your custom queryset subclass (you can check this by yourself by inspecting your queryset type), but the edit_desc() method has not been called on them.

Technically you could "solve" this by overridding all the exclude/filter/ect methods to reapply your edit_desc() method, but it will be terribly inefficient (even more than it actually is), so you may want to rethink about what your real problem is and how to solve it in a more efficient and less intrusive way. Perhaps explaining your concrete use case could lead to better answers at how to solve it ?

EDIT : given your comment, a possible (and much more efficient) solution would be to override the QuerySet parts that yield or return a model instance (or just the raw values) so you can process your model instance / data at this point. You may want to have a look at django/db/query.py, specially the ModelIterable and ValueIterable classes. That's a lot more work than your current solution but well...

Another possibly simpler solution if you only care about model instances (not raw data) could be to use a ModelProxy, overridding it's __init__ method to add your processing at this point...

edited Dec 21 '17 at 14:42

answered Dec 21 '17 at 13:33

bruno desthuilliers

75,974
6
88
118

Bruno, thanks for your answer. This makes me sad. The point was that objects has different versions with serialized data and I wanted to return default or custom manager with specific versions of the objects regarding to user choice, but without actually reverting whole system`s data to that sate. I`m using django-reversion for this purposes, however this and all similar packages can only revert whole object state to spcific version. So instead I was deciding to create custom manager that will take data from serialized structure and return it a queryset without actually storing changes – Michael Dec 21 '17 at 14:13
Well, you might still be able to implement your feature by subclassing `QuerySet`, but doing it the "right" way (well, the one I can think of) might require a bit more work - cf my edited answer. – bruno desthuilliers Dec 21 '17 at 14:42
Oh that has a sense, especially ModelProxy seems could be a nice solution. Do I understan correctly, that ModelProxy adds some specific functionality for model, without actually changing data in db? – Michael Dec 21 '17 at 15:11
1

A ProxyModel is mostly "another model using the same table" (as the model it proxies). And yes, the point is to have a different behaviour on the same dataset. wrt/ the "without actually changing data in db", a ProxyModel is a model, so if you save it to the db, well you do update your db, so it's up to you to make sure you won't mess your data at this point - but you'd have the same problem with your current solution. – bruno desthuilliers Dec 22 '17 at 08:26

score 0 · Accepted Answer · edited Oct 19 '19 at 14:21

Finally I realized, that queryset methods not actually work with a queryset object. Instead, they add some extra SQL syntax and perform a query again to the database.

So, changing object`s data without saving it, and then trying to perform some filtering or etc will cause to querying the database again and retrieving initial data.

If someone has similar use-case as in my question, unfortunately there is a need in another approach. Possible ones are:

Redefining queryset methods to make them work at queryset locally, as with list (not preferred, can cause to various issues in default django`s behaviour)
Creating extra model, that will be logically related to main model (possibly may be fine)

Anyway, thanks a lot for @bruno desthuilliers and his suggestions, if someone has some better ideas, will be happy to see them in comments.

Django custom manager`s method that changes data works incorrectly with other manager`s methods (filter, get, etc.)

2 Answers2