0

I am struggling with how best to perform a long computation for my web app. Basically I need to download and parse a very large remote dataset (~500mb, ~150mb zipped) every day.

I originally thought to just schedule it as a cron job and load a frontend instance... the problem I was thinking is that this instance is so much more computationally intensive than anything else my app does, so it seems wasteful to increase the frontend memory limit above 128mb just for this one step.

I was looking into running it as a backend, but it doesn't seem like there is a straightforward way to load a backend on a schedule like a cron job (I guess cron.yaml only loads frontend instances?)

I guess I can use task queues, but can I schedule it as a task queue and start it with cron.yaml?

I hope my questions make sense -- I appreciate any help!

user3058197
  • 1,052
  • 1
  • 9
  • 21

1 Answers1

2

I guess cron.yaml only loads frontend instances?

The 'target' part of the cron config can set to the name of a backend to have the code run there.

but can I schedule it as a task queue and start it with cron.yaml?

Yes. You can have a cron task (running on a frontend) that schedules a task that will run on a backend. The benefit of this over targetting the backend via cron directly, is that you can configure the retry behaviour of tasks, but can't do so for crons.

Greg
  • 10,350
  • 1
  • 26
  • 35
  • Thanks for the reply -- I'm digging through the Google App Engine documentation and I can't find the methods for pulling up a backend instance from a frontend script... do you by chance know what they are (or can you link me to them)? Thanks! – user3058197 Dec 31 '13 at 22:16
  • 1
    Assuming python and the taskqueue API (rather than deferred): https://developers.google.com/appengine/docs/python/taskqueue/overview-push#Python_Push_queues_and_backends – Greg Jan 01 '14 at 00:34