1

I was having a concurrency issue (running multiple pipelines at the same time) and hence would like to run my exported pipelines (json) directly from dataproc. Is there any tutorial or a script to help me do so?

Any help or leads would be appreciated

Sharath Nair
  • 11
  • 1

1 Answers1

0

If your pipeline contains a dataproc job, you will need to extract that job info and create a dataproc job create request.

There is no feature to extract only dataproc jobs from a datafusion pipeline.

Eduardo Ortiz
  • 715
  • 3
  • 14