0

Question: How to create a lightweight on-demand instance, preconfigured w/ Java8 and my code, pull a task from a task queue, execute the memory-intensive tasks, and shut itself down. (on-demand, high memory, medium cpu, single task executors)

History: I was successfully using Google App Engine Task Queue in Java for "bursty" processing of relatively rare events - maybe once a week someone would submit a form, the form creates ~10 tasks, the system would chew up some memory and CPU cycles thinking about the tasks for a few minutes, save the results, and the webpage would be polling the backend for completion. It worked great within Google App Engine - Auto scaling would remove all idle instances, Task Queues would handle getting the processing done, I'd make sure not to overload things by setting the max-concurrent-requests=1, and life was good!

But then my tasks got too memory intensive for instance-class: F4_1G I'd love to pick something with more memory, but that isn't an option. So I need to figure something out.

I think my best bet is to spin up a generic instance using the API com.google.api.services.compute.model.Instance but get stopped there. I'm so spoiled with how easy the Task Queue was to build that I'd hate to get lost in the weeds just to get the higher memory instance - I don't need a cluster, and don't need any sort of reliability!

  1. Is this a docker container thing?
  2. Is it going to be hard auth-wise to pull from the Pull Queue outside of GAE?
  3. Is it crazy to spin up/down an instance (container?) for each task if a task is ~10 minutes?

I found some similar questions, but no answers that quite fit:

Benjamin H
  • 5,164
  • 6
  • 34
  • 42
  • 1
    Have you looked at App Engine [Flexible environments](https://cloud.google.com/appengine/docs/flexible/java/configuring-your-app-with-app-yaml)? It's a GCE/GAE hybrid so you get auto scaling of App Engine with the flexibility of Compute Engine. It's *still* beta though. – tx802 Aug 17 '16 at 06:11
  • I really liked that App Engine Standard scaled down to 0 - if I wasn't using any task workers, it would go idle and not charge me for uptime. https://cloud.google.com/appengine/docs/flexible/java/configuring-your-app-with-appengine-web-xml#auto-scaling – Benjamin H Aug 19 '16 at 02:36
  • I'd scratch off the requirement list the **single task** execution - doesn't fit well neither with GAE nor with GCE pricing. Once you spin-up an instance might as well let it handle **all** tasks and requests related to an isolated event (like your form) before shutting it down explicitly (GAE's dynamic instances are even more attractive as you don't even need to worry about explicitly shutting down the instances). – Dan Cornilescu Aug 28 '16 at 17:17

1 Answers1

2

I would have a read about GAE modules. These can be set to use basic scaling so an instance gets created on demand, then expires some time later, set by you in your appengine-web.xml using something such as:

<basic-scaling>
 <max-instances>2</max-instances>
 <idle-timeout>5m</idle-timeout>
</basic-scaling>

If the module processes requests from a task queue then is has 10 minutes to get its job done, which is probably ample for many tasks.

andrew pate
  • 3,833
  • 36
  • 28