2

I am trying to create a data pipeline in Databricks using Workflows UI. I have significant number of tasks which I wanted to split across multiple jobs and have dependencies defined across them. But it seems like, in Databricks there cannot be cross job dependencies, and therefore all tasks must be defined in the same job, and dependencies across different tasks can be defined. This is resulting in a very big and messy job diagram.

Is there any better way here ?

P.S. I have access only to the UI portal, won't be able to execute Jobs API ( if there is some way to do this is via API )

Abhishek
  • 83
  • 10

1 Answers1

0

It's possible to trigger another job but you will need to use REST API for that, plus you will need to handle it's execution, etc.

But ability to have another job as a subtask is coming - if you watch recent quarterly roadmap webinar, you will see a slide about "Enhanced control flow" that mentions "Trigger another job" functionality.

Alex Ott
  • 80,552
  • 8
  • 87
  • 132
  • It is interesting, that it was mentioned in the roadmap webinar, but you can hardly find any information about it now. Thanks for your hint here. ;) – chAlexey May 11 '23 at 09:14
  • Roadmap is just pointing that it will happen in the future, documentation is added only when feature is available in the public preview – Alex Ott May 11 '23 at 09:58
  • I found the video on vimeo about it: https://vimeo.com/793497319 (it is at 39:54). It was also mentioned in the last roadmap presentation: https://vimeo.com/825719263 (it starts at around 12:07). It is coming in this quarter. – chAlexey May 22 '23 at 14:44