10

I wonder if anybody has encountered the strange problem on Google App Engine's NDB: after creating a new entity and saving it by put(); and then query() immediately, there is always one less item. For example,

class Item(ndb.Model):
    ...
    ...

items = Item.query().fetch()
length1 = len(items)
item = Item()
item.put()
items = Item.query().fetch()
length2 = len(items)

In the above, length1 is always equal to length2. However, length2 will be corrected when revisiting the same HTML page later. What is the problem? Thanks.

Randy Tang
  • 4,283
  • 12
  • 54
  • 112

5 Answers5

21

This is expected behaviour; your queries above are only eventually consistent. That is, you're not guaranteed to get the very latest results when querying.

You can get around this by using an ancestor query (see the link above). For your example, you'd need to give each of your items a parent entity and then use Item.query().ancestor(theParentEntity).fetch().

Jesse Rusak
  • 56,530
  • 12
  • 101
  • 102
  • Thanks a lot, Jesse. But `Item` is a root entity. To get around this, I think I need to create a dummy parent model. Is this correct? – Randy Tang Mar 17 '13 at 13:35
  • 1
    Yes, that's right, but note that there are performance implications of having a parent entity. (i.e. you can only have ~5 writes per second to a given entity group) – Jesse Rusak Mar 17 '13 at 13:38
  • I created a dummy parent model: `class DummyParent(ndb.Model): dummy = ndb.StringProperty()` And then create an entity: `dummyParent = DummyParent(id='1') dummyParent.put()`, followed by: `item = Item(parent=dummyParent)`. Now I got an error message: `BadValueError: Expected Key instance, got DummyParent(key=Key('DummyParent', '1'))`. What was the problem? – Randy Tang Mar 17 '13 at 13:54
  • Try `item = Item(parent=dummyParent.key)` – Jesse Rusak Mar 17 '13 at 14:33
  • 1
    Well, there was an error message about the `Item.query().ancestor(dummyParent).fetch()` statement. **TypeError: 'NoneType' object is not callable** – Randy Tang Mar 18 '13 at 05:10
  • Sorry, that should be `Item.query(parent=dummyParent.key).fetch()` – Jesse Rusak Mar 18 '13 at 12:26
  • Still no luck. Error message: **TypeError: __init__() got an unexpected keyword argument 'parent'** – Randy Tang Mar 19 '13 at 23:36
  • Sorry, it's `parent` in other model methods; it's call `ancestor` in this one. You can see it in the [query docs](https://developers.google.com/appengine/docs/python/ndb/modelclass#Model_query). – Jesse Rusak Mar 20 '13 at 00:48
  • how would you suggest approaching this with an existing schema/database? seems like the dummy root element solution requires migrating the data that is already stored. is there a conceptual problem with a wait-loop after the write? not pretty but time is of the essence – ohadpr Aug 26 '13 at 08:37
  • So far as I know, there's no way to have a "wait loop"; entity groups are *the* way to get consistent results from queries. You would have to migrate your data. – Jesse Rusak Aug 26 '13 at 18:22
  • The dummy ancestor approach is indeed the approach I have used - but does it strike anyone else as something of a hack? You'd think that this is a flaw in the design - because it makes an extremely common case - that of access of a model immediately after creation - unnecessarily obfuscated. – BooTooMany Feb 13 '15 at 18:38
8

As @JesseRusak said, you need a Dummy ancestor to solve this little problem (I had the same problem that you recently).

But I didn't make a new DummyEntity, just only a DummyKey for a DummyAncestor. Try this for your problem:

class Item(ndb.Model): ... ... items = Item.query(ancestor=ndb.Key(Item, 'Items')).fetch() length1 = len(items) item = Item(parent=ndb.Key(Item, 'Items')) item.put() items = Item.query(ancestor=ndb.Key(Item, 'Items')).fetch() length2 = len(items)

At least in my case, the DummyAncestor worked.

Tony Wickham
  • 4,706
  • 3
  • 29
  • 35
Eagle
  • 978
  • 1
  • 14
  • 27
1

You could refer to their tutorial on nbd here.
They use a function to generate an ancestor key based on some Property of their ndb model. Depending on how you want your database, you could use a Property that is unique for several items, such as a User property, in a databse in which each user has several post let say. Or you could add a new dummy Property, say dummy = ndb.StringProperty() and for each item initialize that dummy with the same string, this way you'll get all entries that can later be filtered.

jh314
  • 27,144
  • 16
  • 62
  • 82
Mihai Varga
  • 88
  • 1
  • 6
1

The problem you are experiencing is that ndb provides "eventually consistant" data. With eventually consistent data, it typically takes ndb a few seconds to update the data so that you can use it later. This works for most applications, but if you need the data immediately after submitting a new entity, you will need "strongly consistent" data.

Google explains the difference in this great article: https://cloud.google.com/appengine/docs/python/datastore/structuring_for_strong_consistency

lswartsenburg
  • 373
  • 3
  • 8
  • >>"The problem you are experiencing is that ndb provides "eventually consistant" data" - You meant Datastore provides eventual consistency. right? – LearnerEarner Jan 26 '18 at 20:11
-3

I solved this by making the appropriate query, creating the new model record, then appending it to my query before returning it. Modifying yours, it looks like: 

class Item(ndb.Model):
    ...
    ...

items = Item.query().fetch()
length1 = len(items)
item = Item()
item.put()

appended_items = list()
for existing_item in items:
    appended_items.append(existing_item)

appended_items.append(item)
length2 = len(appendeditems)

In this case, appended_items has your query, plus the new element. The list generation is inefficient, but I'm a python/ndb noob and there's probably a way to get the Collection right out of the Query model, which would be much better.

TheRyanBurke_AWS
  • 532
  • 5
  • 11
  • 2
    The code as written could produce duplicates. (It's possible the new item is actually returned in the query result.) It's also not going to work correctly if your query requires sorting/filtering/etc. – Jesse Rusak Sep 26 '14 at 12:56