Exceeded soft memory limit of 512 MB with 532 MB after servicing 3 requests total. Consider setting a larger instance class in app.yaml

Question

We are on Google App engine standard environment, F2 instance (generation 1 - python 2.7). We have a reporting module that follows this flow.

Worker Task is initiated in a queue.

    task = taskqueue.add(
            url='/backendreport',
            target='worker',
            queue_name = 'generate-reports',
            params={
                "task_data"     : task_data
            })

In the worker class, we query Google datastore and write the data to a Google Sheet. We paginate through the records to find additional report elements. When we find additional page, we call the same task again to spawn another write, so it can fetch the next set of report elements and write them to Google sheet.

in the backendreport.py we have the following code.

 class BackendReport(): 
        # Query google datastore to find the records(paginated)  
        result = self.service.spreadsheets().values().update(
                spreadsheetId=spreadsheet_Id, 
                range=range_name,
                valueInputOption=value_input_option, 
                body=resource_body).execute()

        # If pagination finds additional records
        task = taskqueue.add(
                url='/backendreport',
                target='worker',
                queue_name = 'generate-reports',
                params={
                    "task_data"     : task_data
                })

We run the same BackendReport (with pagination) as a front end job (not as a task). The pagination works without any error - meaning we fetch each page of records and display to the front end. But when we execute the tasks iteratively it fails with the soft memory limit issue. We were under the impression that every time a task is called (for each pagination) it should act independently and there shouldn't be any memory constraints. What are we doing wrong here?

Why doesn't GCP spin a different instance when the soft memory limit is reached - automatically (our instance class is F2). The error message says soft memory limit of 512 MB reached after servicing 3 requests total - does this mean that the backendreport module spun up 3 requests - does it mean there were 3 tasks calls (/backendreport)?

score 1 · Answer 1 · answered Mar 10 '20 at 21:58

1

Why doesn't GCP spin a different instance when the soft memory limit is reached

One of the primary mechanisms for when app engine decides to spin up a new instance is max_concurrent_requests. You can checkout all of the automatic_scaling params you can configure here:

https://cloud.google.com/appengine/docs/standard/python/config/appref#scaling_elements

does this mean that the backendreport module spun up 3 requests - does it mean there were 3 tasks calls (/backendreport)?

I think so. To be sure, you can open up Logs viewer, find the log where this was printed and filter your logs by that instance-id to see all the requests it handled that lead to that point.

answered Mar 10 '20 at 21:58

Alex

5,141
12
26

Thanks Alex. There is nothing in the documentation that talks about params to scale based on soft memory limits. How do you recommend we resolve the above issue? – Jack tileman Mar 10 '20 at 22:33
I dont believe app engine provides such a feature. It sounds like you'd need to set `max_concurrent_requests` to `1`, but 512 MB for only 3 requests is alot. Does that sound right to you? There may be improvements to your process that you could do, like working with chunks of a file at a time – Alex Mar 10 '20 at 23:00
I agree 512 MB for 3 requests is a lot. Hence I am befuddled on how to solve this. Each individual task only deals with 10 fields of data. There are no large strings associated in any of these fields. Is the way I am calling the tasks iteratively wrong? – Jack tileman Mar 10 '20 at 23:56
I dont think the issue is the overhead for the tasks. I have a cron job that fans-out many google cloud tasks and I havent hit memory issues. I think the issue is whatever is happening in the tasks. Another thing to consider is there are some python libs that require suck up an obscene amount of memory when they get loaded & used. – Alex Mar 11 '20 at 17:35

score 0 · Answer 2 · answered Mar 11 '20 at 04:19

you're creating multiple tasks in Cloud Tasks, but there's no limitation for the dispatching queue there, and as the queue tries to dispatch multiple tasks at the same time, it reaches the memory limit. So the limitations you want to set in place is really max_concurrent_requests, however not for the instances in app.yaml, it should be set for the queue dispatching in queue.yaml, so only one task at a time is dispatched:

- name: generate-reports
  rate: 1/s
  max_concurrent_requests: 1

Exceeded soft memory limit of 512 MB with 532 MB after servicing 3 requests total. Consider setting a larger instance class in app.yaml

2 Answers2