0

Is it possible to achieve interoperability between a scheduler and a pub/sub in the Google Cloud, so that a task is triggered after a specific time every day, but only if a message arrives?

UPDATED:

Example would be a task scheduled for 10:00 am waits for a msg (a pre-requisite).

  1. At 10:00 the msg has not arrived. The job is not triggered. The msg arrives at 11:00. The job is triggered. (It can then send a msg to start the task to be executed)
  2. At 09:00 the msg arrives. The job is not executed. At 10:00 the job is triggered.
  3. The msg never arrives. The job is never executed.
Galya
  • 6,294
  • 6
  • 27
  • 45
  • Yes, you can create a simple Cloud Function that subscribes to a Pub/Sub topic. Then, create a Pub/Sub topic to trigger that function. After this, create a Cloud Scheduler job that invokes the Pub/Sub trigger. Finally, run the Cloud Scheduler job. You can add any logic into your Cloud Function. You have a good tutorial [here](https://cloud.google.com/scheduler/docs/tut-pub-sub) – sllopis Dec 06 '19 at 11:47
  • @sllopis, actually I'm asking for a little bit more complicated case. You're describing Scheduler job -> sending msg -> triggering Function. I'm looking for: msg -> enabling Scheduler job -> triggering Function after the specified time. I'll update the question with an example – Galya Dec 06 '19 at 12:06

2 Answers2

1

Google's recommended practise is to use Google Cloud Composer such tasks.

You can use cloud composers for variety of use cases including batch processing, real-time / stream processing and cron job / scheduled task style processing.

https://cloud.google.com/composer/

Under the hood Composer is running Apache Airflow over managed GKE cluster. So it's not only orchestration tool but it also gives ability to run code using DAGs (which is essentially a cloud function). Have a look at some example DAG triggers below:

https://cloud.google.com/composer/docs/how-to/using/triggering-with-gcf

So essentially if you create a conditional DAG trigger then it should do the trick.

Hope this helps.

Parth Mehta
  • 1,869
  • 5
  • 15
  • The Composer thing looks promising although I see it more like Orchestrator of the Cloud environment itself. Can you add some details how my example can be achieved with it – Galya Dec 06 '19 at 12:17
  • added some context and link with examples in the original comment. – Parth Mehta Dec 06 '19 at 12:29
1

Your puzzle seems to be an excellent match for using Cloud Tasks. At a high level, I would imagine you writing a Cloud Function that subscribes to the topic that is being published upon. The Cloud Function would contain your processing logic:

  1. Received after 10:00am, run your job immediately.
  2. Received before 10:00am, use Cloud Tasks to post a a task to run your job at 10:00am.

... and that's it.

Kolban
  • 13,794
  • 3
  • 38
  • 60