Hard limit of 10 running tasks per instance in AppEngine queue

Question

I have a GAE project, connected with an App Engine Queue (formerly 'push queue'), which should handle multiple tasks at once (target would be ~30 tasks per running instance). Somehow, I'm facing a hard limit of 10 * [COUNT OF RUNNING INSTANCES] where I'm unable to get beyond. Since I tried several different settings and approaches, I hope for you guys shedding some light on this.

TL;DR

GAE / queue setup for long-running tasks, written in PHP with Symfony, using backend instances (required), facing hard-limit of just 10 running tasks per instance. Any hint?

The use-case

My original use-case involves the task handler to fetch data from an API, processes and stores it to BigQuery. In average, a single task handler is doing ~1.500 requests per minute. For scale, I'd wish to have something like 10 running instances (instance class B4_1G) with ~300 running tasks in parallel).

I created a small repository, available here on Github, where I simplified my use-case. For that, I took the plain example code from Google itself. Please read the step-by-step guide for how to reproduce my issue.

Dependencies

The tasks are running rather long (~15 minutes each). Because of the limitation of GAE Frontend instances being cut at 10 minutes, I'm using Backend instances (also, for getting more memory capacity).

The problem

Generally speaking, I'm unable to get more tasks running than the mentioned 10 * [COUNT OF RUNNING INSTANCES]. I tried a lot (see below).

Running my original code in production, I get to about 30 % or memory usage per instance (if using B4). In other words: Speaking of resources, an instance would be capable of handling way more tasks, but it leaves up to 70% RAM unused.

Just 'something' restricts me to this hard limit of 10. I have no clue, where this limit could be. Maybe it's just one minor flag somewhere. :-(

What I already tried

All of that, obviously, without the effect of getting over the limitation.

• Tried my example code in a brand new project (and even account)
• Contacted Google Cloud support (lots of communication, no real solution)
  • They do not believe in such a hard limit
  • Most viable information was: Need to define max_concurrent_requests for the GAE itself.
  • But that's not possible for basic_scaling thus for Backend instances.
  • Even with Frontend instances, I was unable to get over the limit.
• Tried running more instances and/or larger instance types (up to B4_1G)
• Tried Frontend instances (not my use-case for production, but just as test)
• Configured GAE with entrypoint and --workers flag (see docs)
  • Tried queue max_concurrent_requests: 15 with GAE --workers=15 for just 1 running instance
• Tried both php72 and php73 environment
• Played around with all parameters of app.yaml and queue.yaml, according to the documentation

Since I'm really out of thoughts, I appreciate any type of hint where to look at. Maybe, you could even run & test my example code on your side.

Thank you very much in advance.