0

I have some sort of memory leakage in my program. I made a previous post but this code-snippet it more streamlined.

It fetches 100 users per loop; we got ~6000 users.

def sync_users():
  api = API()#class handling api-request - no leak
  maxResults = 100
  page_token = ""
  user  = None
  users = None
  while True:
    gc.collect()
    logging.debug(memory_usage().current())
    users = api.user_list(maxResults=maxResults, pageToken=page_token)
    if "users" not in users:
      break
    for user in users["users"]:
      deferred.defer(sync_user, user)
    if "nextPageToken" in users:
      page_token = users["nextPageToken"]
    else:
      break

I ran it with a paused Task-queue so that no other task ran in parallel. This is my memory-log. 

    92.703125
    92.70703125
    93.81640625
    94.80859375
    96.2109375
    97.3515625
    98.75
    99.71484375
    101.1796875
    102.44140625
    103.5
    ...
    150.25390625
    151.2421875

Memory usage grows linearly and ends with: Exceeded soft private memory limit of 128 MB with 153 MB after servicing 5586 requests total

Sometimes it manage to finish before memory runs out; it's not an infinite loop.

Community
  • 1
  • 1
Niklas Ternvall
  • 510
  • 1
  • 3
  • 19
  • Why did you create a new post instead of editing the existing one? – Андрей Беньковский Jan 27 '16 at 15:03
  • Possible duplicate of [Memory leak in Google ndb library](http://stackoverflow.com/questions/33036334/memory-leak-in-google-ndb-library) – Alex Jan 27 '16 at 15:06
  • Since you are using a deferred execution, I think that your use objects will still be references until the deferred has completed the execution. So the garbage collection will depend on the runtime of deferred thus on the network. – Simone Zandara Jan 27 '16 at 15:10
  • I wish i had known earlier that a comment like `class handling api-request - no leak` prevents memory leaks. But seriously, if anyone is supposed to help i doubt it will work without looking at your API class. – konqi Jan 27 '16 at 15:34
  • Made new post so old comments would make sense. I have checked the size of the API-object and it's the same. – Niklas Ternvall Jan 27 '16 at 15:39
  • @SimoneZandara Do all tasks share memory making deferred.defer the reason for this issue? – Niklas Ternvall Jan 27 '16 at 15:46
  • @NiklasTernvall I am no expert in GAE, so you should double check, but deferred is either a new thread or an asyncio routine which share of course the memory with the main execution thread. – Simone Zandara Jan 27 '16 at 15:53
  • Deferred only submits a task to a task queue. I am still not convinced there is a memory leak. You may find you just need more memory than is available in the instance size. Unless you can complete the request, and allow cleaning up (gc) post request, and then find repeated requests continue to grow the memory used, I will have doubts. Your problem is the request handler may still hold references, which can't be gc'd until the request completes. – Tim Hoffman Jan 28 '16 at 09:45

0 Answers0