0

I been trying to understand how I can setup the follow architecture on Google's cloud:

  • Google app engine receives HTTP request
  • Google app engine queues a pull task as a result of the HTTP request
  • The task is received by a auto scaling google compute engine instance group

Diagram from google: Architecture: Digital Asset Management and Sharing Are there solutions that someone can point me to how to setup a auto scaling task pull task queue handler? Each of my tasks will take approximately a minute to process I estimate.

Stan Wiechers
  • 1,962
  • 27
  • 45

1 Answers1

1

GCE has a new feature called autoscaler that you can use to automatically scale a group of GCE instances, the documentation is here GCE Autoscaler. It should work well for this use case.

There's also a REST API for pull queues where you can pull from outside of app engine Pull Queue. That will let you pull tasks from the queue using compute engine.

Matt J
  • 423
  • 3
  • 6
  • So assume: My code is running a loop that polls for new tasks, creates a new thread for each task and the auto scaler uses CPU utilization as a measure for scaling up/down. Question: If new instances are added with my setup I can't really control that the least busy ones are getting the new tasks right? Is there a load balancer for this? – Stan Wiechers Dec 20 '14 at 19:52
  • There isn't really a load balancer for that, but it should still work because your worker instances will pull tasks as they are ready to process them. Heavily loaded instances will pull slower and new instances will pull faster until they're more or less equal. – Matt J Dec 29 '14 at 17:35