Google App Engine Datastore: Dealing with eventual consistency

Question

I am developing a GAE web application and I need to create and remove associations among instances of two Entities which have no ancestor relationship (also consider the same instance may have multiple associations that may vary in time while the ancestor relationship, once created, can't be removed). I have experienced the 'eventual consistency' policy which means data in my web page are not refreshed coherently with the relationships I am creating/removing. However I have seen that by executing the put() method twice the consistency seems to be force.

This is compliant with the "Eventual Consistency" definition which states that "... provided no new updates are made ..." data will eventually be consistent and, since I am making another update (actually the same two times) I guess the system forces the consistency.

Am I right? Is there any other more elegant way of doing it?

Update: My AIM

Consider I have a list of entities of type A, it doesn't matter what they actually represent, let's say they represent the main entities of my business I have a list of another entities of type B representing services the entities of type A can rely on. it is a many to many relationship so A service b can be used by many entities a of type A An entity a can be served by many services b of type B

I have a web page allowing me to create such relationship, (a Jinja2 template with a Jquery Ajax callback from the client side and a webapp2 python request handler relying on the datastore in the server side). When a relationship is removed from the datastore I refresh the data by making another query on the a entity and depicting the list of b keys it has related. In this case I still see the removed b.key in the list of b keys related to a. Which is not what I would expect.

UPDATE: SOME CODE

here is the model

class A(ndb.Model):
    name = ndb.StringProperty()
    services = ndb.KeyProperty(repeated=True)

class B(ndb.Model):
    name = ndb.StringProperty()
    servedEntities = ndb.KeyProperty(repeated=True)

here is the code I use to create the relationship

                a.services.append(b.key);
                b.servedEntities.append(a.key);
                a.put()
                b.put()

here is the code I use to remove the relationship

               a.services.remove(b.key);
               b.servedEntities.remove(a.key);
               a.put()
               b.put()

There is no ancestor relationship between a and b (and there can not be) After the relationship removal I if retrieve again a from the Datastore I can still see b.key listed in a.services

You cannot force it by doing a put twice, its useless. Can you describe why do you need them to be consistent? — Dmytro Sadovnychyi, Dec 24 '14 at 08:30
If you will include minimum possible code sample, which represents the problem -- would be much easier to help you. — Dmytro Sadovnychyi, Dec 24 '14 at 08:40
It is visible only when you doing a query, right? if you will try `a.key.get()` -- `b` in sot listed there? — Dmytro Sadovnychyi, Dec 24 '14 at 08:48
there's also a per-request cache which may be being hits I can't remember off the top of my head what the rules are but it might explain this. — Paul Collingwood, Dec 24 '14 at 11:31
You should at least show the models you are using. What are `services` and `servedEntities`? And please show exactly how you are retrieving the entities again after updating. — Daniel Roseman, Dec 24 '14 at 11:36

score 1 · Answer 1 · answered Dec 24 '14 at 17:19

The answer to your question lies in this statement:

When a relationship is removed from the datastore I refresh the data by making another query on the a entity.

Why do you need a new query?

Let's say a user subscribed to services x, y and z. Now a user tells you to drop service z from the list. You go to the Datastore and make the necessary change. Instead of running a new query, however, which may still show z among the returned entities, you can simply remove z from your user entity on the client side and update the UI accordingly.

This is obviously a simplified example. I faced a similar challenge when I wrote the school scheduling app, which had a more complex use case - a single change may affect many entities in the selected time period (e.g. "schedule a lesson every school day until the end of the quarter"), so I was tempted to simply run a new query to refresh the schedule view. Obviously, I ran into the eventual consistency problem magnified by the fact that sometimes dozens of entities had to be created. Then I realized that I already have all the data that I need to refresh the view without running a new query. It's a bit more code, but it's a clean and reliable solution.

Ok but if the user refreshed the page he would see the old item coming back again — lowcoupling, Dec 25 '14 at 18:59
Considering the speed of user fingers and the network latency, the refreshed page will be correctly updated unless a user somehow managed to hit a refresh button within milliseconds of hitting the save button. How often does that happen? Do you want to double your write costs to account for this possibility? (In my experience, eventual consistency rarely takes more than a second to resolve itself even when multiple entities are involved). — Andrei Volgin, Dec 25 '14 at 19:04
@lowcoupling Since this is an ajax-y app, refreshing is less common. You can also [leverage memcache][1] to minimize this happening. But, in the end, if you NEED this kind of consistency, your choices are either 1) restructure your schema to use entity groups or other strongly consistent features of the datastore or 2) don't use app engine. https://cloud.google.com/datastore/docs/articles/balancing-strong-and-eventual-consistency-with-google-cloud-datastore/#h.kzvtug7i4vlw — Chris, Jun 15 '15 at 19:20

Google App Engine Datastore: Dealing with eventual consistency

1 Answers1

Linked