3

We have a long-time running service on Google App Engine(GAE). However, after a while, we began to get the error in the image. No changes to the code. I don't understand why we're having the problem we didn't have before.

2020-01-05 08:31:32.704 UTC-8 Exceeded soft memory limit of 2048 MB with 2068 MB after servicing 0 requests total. Consider setting a larger instance class in app.yaml.

2020-01-05 08:31:32.705 UTC-8 This request caused a new process to be started for your application, and thus caused your application code to be loaded for the first time. This request may thus take longer and use more CPU than a typical request for your application.

2020-01-05 08:31:32.705 UTC-8 The warmup request failed. Please check your warmup handler implementation and make sure it's working correctly.

Error image

Edit: System Info;

We are running python. I have cronjobs.

Community
  • 1
  • 1
sumeyyeemir
  • 215
  • 2
  • 3
  • 13
  • This means that there's a memory leak happening. Can you tell me a bit more about the systems you are using? Are you running Java/Python? If you are using ndb, loading quite a lot of entities in a single query fetch will cause the memory limit to exceed. Or you might be loading quite some chunks of data from a third party system. (Keeping all the data loaded in memory). – yaswanth Jan 06 '20 at 04:51
  • We are running python. Actually, I checked to see if there was too much data loading. I deleted the cache of the data taken from the database. I even set up another new service in a different structure. The same error always occurs. I can't find the source of this error that doesn't normally come. Could it be due to a new operation of Google? @yaswanth – sumeyyeemir Jan 06 '20 at 06:56
  • @sumeyye as yaswanth said, it's a memory issue, you probably have a memory leak somewhere, not related to google. Carefully review your application with a profiler, check memory usage and try to understand what you can do to fix it or if you need an instance with bigger memory – Pievis Jan 06 '20 at 10:26
  • If you are using the "inbound_service: warmup" try remove it – Doug Jul 13 '20 at 03:53

4 Answers4

2

Do you know, how much memory it consumes when you run it locally?

With Stackdriver Profiler which is in beta, you can find what's happening with your memory usage.

When there is the Exceeded soft private memory limit error, you can follow two alternatives:

1) You can upgrade your instance to an another with more memory

2) You can reduce the chunks of data you process in each request, process the XML file in smaller pieces and keep the smaller instance doing the work.

Here is a similar post which can help.

Nibrass H
  • 2,403
  • 1
  • 8
  • 14
2

memory usage

I am experiencing the same increasing memory (memory leak?) issue on GAE standard evironment, python3 + flask on projects that were running for years without any problem. I also tried instances with higher memory, but that was not a solution.

Switch from requests_futures library with requests and explicity set entrypoint: gunicorn -b :8080 -w 1 main:app solved issue to me in one of the project.

Seems that on python2 these problems do not occur.

janq
  • 21
  • 2
  • Actually, we solved this problem by switching to kubernetes. We really started to have troubles in Google-App-Engine. Instead of wasting time on it, it was smarter to switch to kubernetes. @janq – sumeyyeemir Feb 03 '20 at 06:50
  • On my projects it only occurs when using python 3. – janq Feb 05 '20 at 05:18
  • Only workaround working to me is to restart gunicorn worker after N requests `entrypoint: gunicorn -b :8080 -w 2 --max-requests=500 main:app` – janq Feb 08 '20 at 07:53
  • @sumeyye I'm curious as to how kubernetes resolved this? Surely it doesn't allow for an application to just continue using more memory over time? Or do you think it resolved memory leaks in the underlying app engine platform? – Mark Feb 11 '20 at 04:38
  • @Mark thought I was answering. Sorry for the delay in response. In fact, we have dealt with the app engine problem. We should not have a problem as a memory. At this point it took our time to meet with such an error. Currently, we run our containers in kubernetes with less memory than the memory in app-engine. – sumeyyeemir Mar 17 '20 at 12:21
0

This is to turn the valuable comment by janq on the answer https://stackoverflow.com/a/60024436/7733418 into an answer of its own:

Only workaround working to me is to restart gunicorn worker after N requests entrypoint: gunicorn -b :8080 -w 2 --max-requests=500 main:app

We also used the -w 1 setting as suggested in the answer itself, because we already had concurrent requests disabled for other reasons:

entrypoint: gunicorn -b :$PORT main:app -w 1 --max-requests=500

The difference in total memory usage was huge after deploying this - it is now around 40% of what it was before, with no other change. I don't see any repercussions from workers restarting. This should be in Google's documentation!

pnico_1234
  • 1
  • 1
-1

Try changing the instance_class from app.yaml file. Which will increase the memory limit according to the assigned class. Refer to below link: https://cloud.google.com/appengine/docs/standard#instance_classes

m4n0
  • 29,823
  • 27
  • 76
  • 89
Swapna Sabu
  • 1