31

I have a model object in Django. One of the methods on the object uses row-level locking to ensure values are accurate, like so:

class Foo(model.Model):
    counter = models.IntegerField()

    @transaction.commit_on_success
    def increment(self):
        x = Foo.objects.raw("SELECT * from fooapp_foo WHERE id = %s FOR UPDATE", [self.id])[0]
        x.counter += 1
        x.save()

The problem is if you call increment on a foo object, the object's values no longer reflect the values in the database. I need a way to refresh the values in the object, or at least mark them as stale so they're refetched if necessary. Apparently, this is functionality the developers of Django refuse to add.

I tried using the following code:

for field in self.__class__._meta.get_all_field_names():
    setattr(self, field, getattr(offer, field)) 

Unfortunately, I have a second model with the following definition:

class Bar(model.Model):
    foo = models.ForeignKey(Foo)

This causes an error, because it shows up in the field listing but you cannot getattr or setattr it.

I have two questions:

  • How can I refresh the values on my object?

  • Do I need to worry about refreshing any objects with references to my object, like foreign keys?

Chris B.
  • 85,731
  • 25
  • 98
  • 139
  • 1
    Why are 'refreshing' your objects? If you are changing the fields intentionally, why aren't you saving them? If you are _not_ changing the fields intentionally, why aren't you re-fetching the data from the db? – Collin Green Mar 20 '12 at 17:53
  • "the object's values no longer reflect the values in the database" which values? Counter will be updated in db and on your instance of the model, no? – AJP Apr 09 '13 at 09:57
  • Ticket #901, which you've linked above, has been re-opened by a core developer, so maybe there's some hope yet. – keturn Nov 22 '13 at 19:00
  • 2
    possible duplicate of [Reload django object from database](http://stackoverflow.com/questions/4377861/reload-django-object-from-database) – Anto Mar 18 '14 at 15:41
  • For those of you still getting old values, Django+MySQL may be getting old values if you are using transactions (enabled by default in Django 1.5). See [How do I force django to ignore caches and reload data](http://stackoverflow.com/questions/3346124/how-do-i-force-django-to-ignore-any-caches-and-reload-data) – Ryan Rapp Mar 29 '14 at 22:04

12 Answers12

48

Finally, in Django 1.8, we have a specific method to do this. It's called refresh_from_db and it's a new method of the class django.db.models.Model.

An example of usage:

def update_result(self):
    obj = MyModel.objects.create(val=1)
    MyModel.objects.filter(pk=obj.pk).update(val=F('val') + 1)
    # At this point obj.val is still 1, but the value in the database
    # was updated to 2. The object's updated value needs to be reloaded
    # from the database.
    obj.refresh_from_db()

If your version of Django is less than 1.8 but you want to have this functionality, modify your model to inherit from RefreshableModel:

from django.db import models
from django.db.models.constants import LOOKUP_SEP
from django.db.models.query_utils import DeferredAttribute

class RefreshableModel(models.Model):

    class Meta:
        abstract = True

    def get_deferred_fields(self):
        """
        Returns a set containing names of deferred fields for this instance.
        """
        return {
            f.attname for f in self._meta.concrete_fields
            if isinstance(self.__class__.__dict__.get(f.attname), DeferredAttribute)
        }

    def refresh_from_db(self, using=None, fields=None, **kwargs):
        """
        Reloads field values from the database.
        By default, the reloading happens from the database this instance was
        loaded from, or by the read router if this instance wasn't loaded from
        any database. The using parameter will override the default.
        Fields can be used to specify which fields to reload. The fields
        should be an iterable of field attnames. If fields is None, then
        all non-deferred fields are reloaded.
        When accessing deferred fields of an instance, the deferred loading
        of the field will call this method.
        """
        if fields is not None:
            if len(fields) == 0:
                return
            if any(LOOKUP_SEP in f for f in fields):
                raise ValueError(
                    'Found "%s" in fields argument. Relations and transforms '
                    'are not allowed in fields.' % LOOKUP_SEP)

        db = using if using is not None else self._state.db
        if self._deferred:
            non_deferred_model = self._meta.proxy_for_model
        else:
            non_deferred_model = self.__class__
        db_instance_qs = non_deferred_model._default_manager.using(db).filter(pk=self.pk)

        # Use provided fields, if not set then reload all non-deferred fields.
        if fields is not None:
            fields = list(fields)
            db_instance_qs = db_instance_qs.only(*fields)
        elif self._deferred:
            deferred_fields = self.get_deferred_fields()
            fields = [f.attname for f in self._meta.concrete_fields
                      if f.attname not in deferred_fields]
            db_instance_qs = db_instance_qs.only(*fields)

        db_instance = db_instance_qs.get()
        non_loaded_fields = db_instance.get_deferred_fields()
        for field in self._meta.concrete_fields:
            if field.attname in non_loaded_fields:
                # This field wasn't refreshed - skip ahead.
                continue
            setattr(self, field.attname, getattr(db_instance, field.attname))
            # Throw away stale foreign key references.
            if field.rel and field.get_cache_name() in self.__dict__:
                rel_instance = getattr(self, field.get_cache_name())
                local_val = getattr(db_instance, field.attname)
                related_val = None if rel_instance is None else getattr(rel_instance, field.related_field.attname)
                if local_val != related_val:
                    del self.__dict__[field.get_cache_name()]
        self._state.db = db_instance._state.db

class MyModel(RefreshableModel):
    # Your Model implementation
    pass

obj = MyModel.objects.create(val=1)
obj.refresh_from_db()
blindOSX
  • 711
  • 8
  • 14
  • just wrote a ticket in the django bugtracker as this seems to break on models that have a foreignkey set to null: https://code.djangoproject.com/ticket/24418 – Johannes Lerch Feb 26 '15 at 14:19
  • you should update the code sample as the bug got fixed: https://github.com/django/django/commit/5cf96b49e43daea6d4a0ba1c80c45e90c74f4e47 – Johannes Lerch Mar 02 '15 at 11:52
  • Updated! Thanks @JohannesLerch for the report! – blindOSX Mar 03 '15 at 16:40
  • This doesn't seem to work with 1.4, any tweaks to get it there? Error I'm getting is AttributeError: 'Options' object has no attribute 'concrete_fields' – RaresMan Oct 05 '15 at 02:30
  • Be aware the built-in `refresh_from_db` refreshes attributes but NOT the attributes of related objects. See https://stackoverflow.com/a/52046068/237091 for a revised implementation that does. – Scott Stafford Aug 27 '18 at 19:48
14

I assume you must need to do this from within the class itself, or you would just do something like:

def refresh(obj):
    """ Reload an object from the database """
    return obj.__class__._default_manager.get(pk=obj.pk)

But doing that internally and replacing self gets ugly...

DrMeers
  • 4,117
  • 2
  • 36
  • 38
  • 12
    Yes, which is why the Django developers should add a "refresh" method. Oddly, they refuse to do it, because they feel it's trivial to do on your own. I can't figure out why they think it's easy. – Chris B. Mar 23 '12 at 13:20
1

Hmm. It seems to me that you can never be sure that your any foo.counter is actually up to date... And this is true of any kind of model object, not just these kinds of counters...

Let's say you have the following code:

    f1 = Foo.objects.get()[0]
    f2 = Foo.objects.get()[0]  #probably somewhere else!
    f1.increment() #let's assume this acidly increments counter both in db and in f1
    f2.counter # is wrong

At the end of this, f2.counter will now be wrong.

Why is refreshing the values so important - why not just can get back a new instance whenever needed?

    f1 = Foo.objects.get()[0]
    #stuff
    f1 = Foo.objects.get(pk=f1.id)

But if you really need to you could create a refresh method yourself... like you indicated in your question but you need to skip related fields, so you could just specify the lists of fieldnames that you want to iterate over (rather than _meta.get_all_fieldnames). Or you could iterate over Foo._meta.fields it will give you Field objects, and you can just check on the class of the field -- I think if they are instances of django.db.fields.field.related.RelatedField then you skip them. You could if you wanted then speed this up by doing this only on loading your module and storing this list in your model class (use a class_prepared signal)

Tim Diggins
  • 4,364
  • 3
  • 30
  • 49
0

I have some long running processes which are running in parallel. Once the calculations are done, I want to update the values and save the model, but I don't want the entire process to tie up a transaction. So my strategy is something like

model = Model.objects.get(pk=pk)

# [ do a bunch of stuff here]

# get a fresh model with possibly updated values
with transaction.commit_on_success():
    model = model.__class__.objects.get(pk=model.pk)
    model.field1 = results
    model.save()
Joseph Sheedy
  • 6,296
  • 4
  • 30
  • 31
0

This combines the best of the two answers above, and adds up-to-date django syntax:

Get fresh data and guarantee* it stays fresh for your transaction:

def refresh_and_lock(obj):
    """ Return an fresh copy with a lock."""
    return obj.__class__._default_manager.select_for_update().get(pk=obj.pk)

This will only work if everything that changes the object goes through select_for_update. Other processes that get the object without a lock will hang at save() instead of get(), and stomp on the change right after the first transaction commits.

Scott Smith
  • 1,002
  • 1
  • 14
  • 19
0

I had a similar need and, whilst you cannot effectively refresh the existing object without potentially tampering its integrity, you can still enforce best practice at implementation time. For what I am concerned, I earmark the object as stale and make that prevent any further access to it, as illustrated in the below example:

class MyModelManager(Manager):
    def get_the_token(self, my_obj):

        # you need to get that before marking the object stale :-)
        pk = my_obj.pk

        # I still want to do the update so long a pool_size > 0
        row_count = self.filter(pk=pk, pool_size__gt=0).update(pool_size=F('pool_size')-1)
        if row_count == 0:
            # the pool has been emptied in the meantime, deal with it
            raise Whatever

        # after this row, one cannot ask anything to the record
        my_obj._stale = True

        # here you're returning an up-to-date instance of the record
        return self.get(pk=pk)


class MyModel(Model):
    pool_size = IntegerField()

    objects = MyModelManager()

    def __getattribute__(self, name):
        try:
            # checking if the object is marked as stale
            is_stale = super(MyModel, self).__getattribute__('_stale'):

            # well, it is probably...
            if is_stale: raise IAmStale("you should have done obj = obj.get_token()")
        except AttributeError:
            pass

        # it is not stale...
        return super(MyModel, self).__getattribute__(name)

    def get_token(self):
        # since it's about an operation on the DB rather than on the object,
        # we'd rather do that at the manager level
        # any better way out there to get the manager from an instance?
        # self._meta.concrete_model.objects ?
        self.__class__.objects.get_the_token(self, my_obj)

(written on the fly, forgive any possible typos :-) )

lai
  • 1,163
  • 10
  • 15
0

You can use

refresh_from_db()

Eg:

obj.refresh_from_db()

https://docs.djangoproject.com/en/1.11/ref/models/instances/#refreshing-objects-from-database

Sarath Ak
  • 7,903
  • 2
  • 47
  • 48
0

I have been using a method like this, because the new builtin refresh_from_db does not refresh children that have had their attributes changed, frequently causing problems. This clears the cache of any foreign keys.

def super_refresh_from_db(self):
    """ refresh_from_db only reloads local values and any deferred objects whose id has changed.
    If the related object has itself changed, we miss that.  This attempts to kind of get that back. """
    self.refresh_from_db()

    db = self._state.db
    db_instance_qs = self.__class__._default_manager.using(db).filter(pk=self.pk)

    db_instance = db_instance_qs.get()
    non_loaded_fields = db_instance.get_deferred_fields()
    for field in self._meta.concrete_fields:
        if field.attname in non_loaded_fields:
            # This field wasn't refreshed - skip ahead.
            continue

        if field.is_relation and field.get_cache_name() in self.__dict__:
            del self.__dict__[field.get_cache_name()]
Scott Stafford
  • 43,764
  • 28
  • 129
  • 177
0

I had the same issue and this worked for me:

obj.reload()
סטנלי גרונן
  • 2,917
  • 23
  • 46
  • 68
Boobo
  • 1
  • 1
0

I see why you're using SELECT ... FOR UPDATE, but once you've issued this, you should still be interacting with self.

For example, try this instead:

@transaction.commit_on_success
def increment(self):
    Foo.objects.raw("SELECT id from fooapp_foo WHERE id = %s FOR UPDATE", [self.id])[0]
    self.counter += 1
    self.save()

The row is locked, but now the interaction is taking place on the in-memory instance, so changes remain in sync.

Chris Pratt
  • 232,153
  • 36
  • 385
  • 444
  • I thought about that, but it doesn't work. Let's say we have two processes, A and B. If process A loads the object, process B can still modify the counter field in the database. If process A now locks the database row but operates on the in-memory instance of the object, it's operating on stale data. – Chris B. Mar 20 '12 at 21:58
  • 1
    If process A and B both use "select ... for update" it will work. Process B will hang until process A commits or rolls-back, then get the updated copy of the counter. – Scott Smith Jan 28 '14 at 19:51
0

You can use Django's F expressions to do this.

To show an example, I'll use this model:

# models.py
from django.db import models
class Something(models.Model):
    x = models.IntegerField()

Then, you can do something like this:

    from models import Something
    from django.db.models import F

    blah = Something.objects.create(x=3)
    print blah.x # 3

    # set property x to itself plus one atomically
    blah.x = F('x') + 1
    blah.save()

    # reload the object back from the DB
    blah = Something.objects.get(pk=blah.pk)
    print blah.x # 4
jterrace
  • 64,866
  • 22
  • 157
  • 202
  • That would work in this instance. In our real problem, however, we need to do more complicated transactions involving a couple of different objects, and it's just a little beyond the reach of F expressions. – Chris B. Mar 20 '12 at 22:17
0

Quick, ugly and untested:

from django.db.models.fields.related import RelatedField

for field in self.__class__._meta.fields:
    if not isinstance(field, RelatedField):
        setattr(self, field.attname, getattr(offer, field)) 

though I think you could do this using some other _meta approach over the isinstance() call.

Obviously, we both know this should be avoided if possible. Maybe a better approach would be to futz with the internal model state?

EDIT - Is the Django 1.4 SELECT FOR UPDATE support going to solve this?

Matt Luongo
  • 14,371
  • 6
  • 53
  • 64
  • The SELECT FOR UPDATE doesn't help, because we still need a way to refresh the object. And I don't think your solution works, because any related fields remain stale. We need to refresh them as well. – Chris B. Mar 23 '12 at 13:18
  • Oh, my mistake- I thought you were only concerned about non-related fields. – Matt Luongo Mar 23 '12 at 20:45