Combine multiple notebooks to run single pipeline

Question

I have 8 seperate notebooks in databricks, so I am currently running 8 different pipelines on ADF, each pipeline contains each notebook and so on. Is there a way to run a single pipeline which runs all notebooks, or is there a way to combine all notebooks into one and use that as master notebook so that it is easier to run pipeline only for master notebook which inturn runs all the notebooks?

Answer 1

You can call all the Notebooks from a parent notebook and call the Parent Notebook from ADF.

Use dbutils.notebook.run("Notebook_name", <time in seconds>, {"Notebook_param1":"value1","Notebook_param2":"value2",...}) in parent to call the particular Notebook.

You can pass the parameters to notebook using the parameters option above.

Sample:

Parent Notebook:

print("Calling Nb1")
print(dbutils.notebook.run("Nb1", 60, {"Nb1_param":"Nb1_value"}))
print("Calling Nb2")
print(dbutils.notebook.run("Nb2", 60, {"Nb2_param":"Nb2_value"}))

Nb1 Notebook:

a=dbutils.widgets.get("Nb1_param")
dbutils.notebook.exit("Nb1_ exit with parameter "+a)

Nb2 Notebook:

a=dbutils.widgets.get("Nb2_param")
dbutils.notebook.exit("Nb2 exit with parameter "+a)

You can return any values from the Notebook to the child if you want.

Call this parent Notebook from ADF.

If you want to pass values to the Notebooks, you can pass the parameters to the Parent Notebook and pass those in the child Notebooks.

Notebook run: