What is using all this memory for a google ndb fetch?

Question

When I fetch 1000 of my models from the datastore, it uses a lot more memory than expected. My datastore indicates that I have >18000 but the total size is 8.31MB

Application 18025 8.31 MB

Here is the code for fetching the entities, with the caches disabled

def memoryForAThousandApplicationsNoCache():
  ndb_ctx = ndb.get_context()
  ndb_ctx.set_cache_policy(lambda key: False)
  ndb_ctx.set_memcache_policy(lambda key: False)
  startUsage = memory_usage().current()
  applications = Application.query().fetch(1000)
  return "Memory usage after applications:  %d MB" % (memory_usage().current() - startUsage)

and I get

Memory usage after applications: 10 MB

What is consuming all this memory? Am I misusing memory_usage?

score 2 · Accepted Answer · answered Mar 20 '18 at 03:53

I'm guessing that you haven't had any reason to profile or dig in to Python memory usage outside of App Engine. Along with the many benefits that come from objects, there's a surprising (to low-level programmers) amount of overhead when representing things as Python objects. And when you're using something like ndb, there are extra (Python) data structures behind the scenes.

Google for "python memory overhead" to find more background. There's a decent graph in this question that may be instructive.

What is using all this memory for a google ndb fetch?

1 Answers1