2

I want to build a task scheduling service on the Google Cloud Platform. The tasks can be as simple as triggering a URL. Tasks can be recurring (once an hour, twice a day, every thursday, ...) and can be created and removed dynamically.

Which services/APIs on the Google Cloud Platform can I use for this?

I have looked into Google App Engine cron jobs but there seems to be no way to programmatically modify them. If possible I would like to avoid running a cron job every minute just to check if there is some task to run.

My framework of choice is ASP.NET Core but if there is a better solution available, e.g. in Java, I'm willing to try it out.

Kirill Rakhman
  • 42,195
  • 18
  • 124
  • 148
  • See my answer here, which recommends using Google Cloud Scheduler https://stackoverflow.com/questions/12116799/adding-dynamic-cron-jobs-to-gae/56230135#56230135 – Chris Halcrow May 21 '19 at 04:00

2 Answers2

2

As you have found out, App Engine Cron Service does not have an API for programmatically managing cron tasks. Cron tasks are configured using a file called cron.yaml and this file can be programmatically modified and uploaded to google cron service(details). I'm not sure about exact requirements for your task scheduling service, bu this could be a good enough solution for your problem.

Another option would be to run a Google compute engine instance. As this is basically a virtual server maintained by you, you will have full control over it; allowing you to choose OS, backend/frontend technologies etc. For example you can run a Linux server, use an asp.net core backend to manage crontab tasks.

Caner
  • 57,267
  • 35
  • 174
  • 180
  • What do you mean about `cron.yaml` file can be programmatically modified? I think it's static – Lin Du Jan 09 '19 at 09:10
  • 1
    @slideshowp2 what I mean is that the file can be updated and re-submitted by the task-scheduling service the question owner wants to implement – Caner Jan 09 '19 at 10:39
  • re-submitted? You mean redeploy the cron tasks after updating `cron.yaml` file? – Lin Du Jan 09 '19 at 10:44
2

I would also recommend to take a look on Google Cloud Tasks. You can create queues and push/pull tasks to/from them:

  • Push queues run tasks by delivering HTTP requests to App Engine worker services. They dispatch these requests at a reliable, steady rate and guarantee reliable task execution. Because you can control the rate at which tasks are sent from the queue, you can control the workers' scaling behavior and hence your costs.
  • Pull queues do not dispatch tasks at all. They depend on other worker services to "lease" tasks from the queue on their own initiative. Pull queues give you more power and flexibility over when and where tasks are processed, but they also require you to do more process management. When a task is leased the leasing worker declares a deadline. By the time the deadline arrives the worker must either complete the task and delete it or the Task Queue service will allow another worker to lease it.

Source: https://cloud.google.com/appengine/docs/standard/java/taskqueue/

Ruben Lopez
  • 704
  • 7
  • 18