1

I'm struggling with a KeyProperty query, and can't see what's wrong.

My model is

class MyList(ndb.Model):
  user = ndb.KeyProperty(indexed=True)
  status = ndb.BooleanProperty(default=True)
  items = ndb.StructuredProperty(MyRef, repeated=True, indexed=False)

I create an instance of MyList with the appropriate data and can run the following properly

cls = MyList
lists = cls.query().fetch()

Returns 

[MyList(key=Key('MyList', 12), status=True, items=..., user=Key('User', 11))]

But it fails when I try to filter by user, i.e. finding lists where the user equals a particular entity; even when using the one I've just used for insert, or from the previous query result.

key = lists[0].user
lists = cls.query(cls.user=key).fetch()

Returns

[]

But works fine with status=True as the filter, and I can't see what's missing?

I should add it happens in a unit testing environment with the following v3_stub

self.policy = datastore_stub_util.PseudoRandomHRConsistencyPolicy(probability=0)                                   
self.testbed.init_datastore_v3_stub(                                                                               
  require_indexes=True,                                                                                      
  root_path="%s/../"%(os.path.dirname(__file__)),
  consistency_policy=self.policy
)
apassant
  • 167
  • 3
  • 11
  • 1
    This is an artifact of Eventual Consistency. It is emulated in the SDK, and you have a couple of choices on how to emulate or not when testing when using the SDK. This is proven by the fact you know you can retrieve the entity, so it exists. Turn off eventual consistancy in the dev environment as another check. But you will have to deal with this in your design. – Tim Hoffman Jun 07 '14 at 00:57
  • @TimHoffman: That's correct, it works fine if I set probability=1, thanks! Yet, I don't understand why the query() w/o parameters works, as well as with another filter (e.g. status=True). Does that mean that, due to Eventual Consistency, indexes are not created yet? I thought the Eventual Consistency state was "all or nothing", not "in-between" like in this case, where only some indexes return the correct data. Is that a bug? – apassant Jun 07 '14 at 06:39
  • Each index may I believe be updated in it's own time. However you are working with the SDK, which at best is an approximation of the real environment. You should write you code to handle the SDK eventual behaviour as that is probably a worst case scenario. If your design can deal with that, then you can deal with production. If on the other hand your design only works if set probability=1 then you are on a slippery slope and will have problems in production. – Tim Hoffman Jun 07 '14 at 07:34
  • @TimHoffman I've already made sure that probablity=0 work for all get(), etc. but that indexing issue was a bit PITA. I've added additional caching so it works fine now. I still think that the different indexing time is confusing (or at least, not well documented). Thanks again! – apassant Jun 07 '14 at 07:55
  • 1
    The fact that indexes are/will be out of date is the crux of eventual consistancy. As you have found get() are always consistent unless there is a failure in googles infrastructure - (we have seen that) However indexes will always have a variable amount of time (normally small) before they are up to date. This applies to deletes as well as writes. – Tim Hoffman Jun 07 '14 at 08:03

2 Answers2

0

user=Key('User', 11) is a key to a different class: User. Not MyList

Perhaps you meant:

user = ndb.KeyProperty(kind='User', indexed=True)
GAEfan
  • 11,244
  • 2
  • 17
  • 33
  • Regarding the different classes, I'm trying to find values where the key is a `User` entity, not a `MyList` one - I've edited the question to make this clear. I also tried with kind='User', same issue – apassant Jun 07 '14 at 00:06
0

Your code looks fine, but I have noticed some data integrity issues when developing locally with NDB. I copied your model and code, and I also got the empty list at first, but then after a few more attempts, the data is there.

Try it a few times?

edit: possibly related? google app engine ndb: put() and then query(), there is always one less item

Community
  • 1
  • 1
mdmoskwa
  • 138
  • 5