2

I am stuck with the problem of memory usage of my FastAPI application.

If I do some load on my application, e.g. 1 req/sec, for a while, e.g. 2-3 minutes, my application will start use 50+mb more than it was after 1 request. I'm using only one end-point for this test and it does the same every time when I call it (some DB calls using google cloud ndb and it runs it in the executor, some gRPC calls using grpcio, some redis calls using aioredis)

I tried to use all memory profilers (guppy3, tracemalloc, objgraph), all of them say that I have a low number of objects when I stopped loading my application e.g. tracemalloc snapshot first lines

#1: /Users/ndavydov/PycharmProjects/project/.venv/lib/python3.8/site-packages/fastapi/utils.py:65: 2707.8 KiB
    return response_field(field_info=field_info)
#2: /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/linecache.py:137: 1922.0 KiB
    lines = fp.readlines()
#3: /Library/Frameworks/Python.framework/Versions/3.8/lib/python3.8/abc.py:102: 1643.4 KiB
    return _abc_subclasscheck(cls, subclass)
#4: /Users/ndavydov/PycharmProjects/project/.venv/lib/python3.8/site-packages/fastapi/dependencies/models.py:38: 425.2 KiB
    self.path_params = path_params or []
#5: /Users/ndavydov/PycharmProjects/project/.venv/lib/python3.8/site-packages/fastapi/dependencies/utils.py:755: 310.3 KiB
    BodyModel: Type[BaseModel] = create_model(model_name)
3265 other: 4269.6 KiB
Total allocated size: 14717.7 KiB

But I can see that my application consumes 275 mb, and after 1-2-3 requests before the load, I saw 190-200 mb of memory usage, but here I can see only 14 mb of allocated objects, and this value isn't changed during load time

I was wondering, if you have some ideas about where I can try to find the root of the issue..

Nikita Davydov
  • 837
  • 7
  • 18
  • Would you please share more of your code? – Life is complex Jul 22 '21 at 15:20
  • Also, how do you see the memory consumption that you mentioned? There is a difference between VIRT, RES, SHR memory. https://itectec.com/ubuntu/ubuntu-what-do-virt-res-and-shr-mean-in-the-top-command/ – Sergei Malanin Jul 27 '21 at 12:44
  • 1
    Related? https://stackoverflow.com/questions/33036334/memory-leak-in-google-ndb-library – AndreFeijo Jul 28 '21 at 00:45
  • @AndreFeijo it might well be. I have tried to track and fix a bug with Flask and Google's datastore library, without success unfortunately: https://github.com/googleapis/python-datastore/issues/84 – BinarSkugga Jul 29 '21 at 02:19
  • @BinarSkugga I'm experiencing the same with my another service without ndb client, but I'm using there gcloud-aio-pubsub + httpx + google cloud tasks with new AsyncClient with AsyncGrpcTransport. It has much less code than in my other service, but the results are the same memory isn't allocated. Again I can't see any problems using all available memory profilers, so only one place where it could be - C-code in grpcio which is used by google cloud core API, I will try to move everything to gcloud-aio-* (because it uses simple REST Google APIs) and return with the results – Nikita Davydov Jul 30 '21 at 13:15
  • I have much fewer libraries in my another service, so I could test everything under real load replacing all google libraries which uses google API core above grpc transport – Nikita Davydov Jul 30 '21 at 13:16
  • 1
    @BinarSkugga Also I've read your issue, and I should add here that I'm issuing a memory leak in service which is deployed to GKE, so it shouldn't be App Engine problem. I advise you to try to load your application locally (e.g 3 rps to the same endpoint) and take snapshots of your memory after 1st request and after 100th-150th request and you probably will see the same - your memory increases to some value and keeps this value even without any requests, but all profilers will say that you have much less size of all objects in your memory – Nikita Davydov Jul 30 '21 at 13:19

0 Answers0