0

I am working on a pipeline using Vertex AI, and I'm facing an issue with scheduling the execution of two components with a delay. Here's the scenario:

  1. Component A generates some data as its output.
  2. Component B processes the output from Component A and sends the result to an external provider for further processing. This external provider typically takes around 2 hours to complete its task.
  3. Component C should be scheduled to run 2 hours after Component B has finished.

My question is, how can I set up this delay between Component B and Component C in the pipeline?

I have considered using the pipeline orchestration capabilities provided by Vertex AI, but I couldn't find a built-in feature for adding such delays between components. I also looked into using Cloud Scheduler, but it seems more suitable for triggering pipeline runs rather than introducing delays within a pipeline.

I would greatly appreciate any insights or suggestions on how to accomplish this delay between the execution of Component B and Component C in the Vertex AI pipeline. Thank you in advance for your help!

pajamas
  • 1,194
  • 1
  • 12
  • 25

2 Answers2

0

Would a new component that does time.sleep for two hours do the job?

Mishka
  • 1
  • As it’s currently written, your answer is unclear. Please [edit] to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers [in the help center](/help/how-to-answer). – Community Jun 25 '23 at 05:01
0

I'd recommend having a separate pipeline for component C. When the component B in the first pipeline completes, publish a message to PuSub and then trigger the Pipeline of Component C with that message in PubSub. For more info: https://cloud.google.com/vertex-ai/docs/pipelines/trigger-pubsub

sim
  • 11
  • 5