2

Is there a way to avoid the provisioning step for subsequently triggered outbound pipelines? It looks like when a pipeline triggers an outbound pipeline, it does the provisioning all over again. Can we simply execute the triggered pipeline on the provisioned cluster of the first?

Thanks.

sacoder
  • 159
  • 13

1 Answers1

0

Cloud Data Fusion has support for running a pipeline against an existing Dataproc cluster, see more details in this doc.

Dagang
  • 24,586
  • 26
  • 88
  • 133