1

I'm using memcache to implement performance gains when reading variables from the datastore. But as it turns out, memcache.get() seems slow.

Here's the code:

def get_settings():
    settings = memcache.get('ds-settings')

I call get_settings() inside a model _from_pb to run some checks everytime there is a read from the datastore, and have measured that 85/90% of the time it takes ~7ms to run, and the other 10%, it takes ~70ms to run. When running this 100, 1k or 10k times inside a loop, it becomes a serious problem.

Any suggestions on how to make reading memcache faster?

Dan McGrath
  • 41,220
  • 11
  • 99
  • 130
user2779653
  • 918
  • 1
  • 9
  • 26

1 Answers1

1

why would you run this in a loop? if you want to multiple keys, batch them by calling memcache.get_multi.

marcadian
  • 2,608
  • 13
  • 20
  • I'm overriding _from_pb to track property changes on my models (and to store versions) like in this [answer](http://stackoverflow.com/questions/21292790/how-to-read-old-property-values-in-a-pre-put-hook). Problem is that _from_pb is called for every object in the query result. I need a global variable (stored in datastore) to activate or disable versions tracking in _from_pb. Module level global variables will no be the same accross instances, so memcache seemed like the solution. But without checking memcache every time, I don't know how to inject this variable in _from_pb. – user2779653 Nov 21 '15 at 02:54
  • For reference, here's [the answer](http://stackoverflow.com/questions/2650014/google-app-engine-persistent-globals) that led me to using memcache for persistent global variables. Just looking for the solution on how to use it inside _from_pb. – user2779653 Nov 21 '15 at 02:57
  • 1
    I think you are taking the wrong approach. I suggest you need to revisit your architecture. By putting this call in _from_pb you are forcing at least 2 rpcs each time you get something. One to retrieve the entity and one for memcache, which may be redundant. Why not store this attribute at the instance level and only check it every n requests, or look for another way. It's difficult to really appreciate your requirement from the description so far. – Tim Hoffman Nov 21 '15 at 06:11
  • I agree, the architecture is wrong. I'm removing this logic from _from_pb and will do this differently. Storing the attribute at instance level doesn't work either, there are too many instances when fetching large lists of objects from the datastore. The only way it seems is to remove these RPC calls from _from_pb. Too expensive. Thanks for your comments! – user2779653 Nov 21 '15 at 18:55