How to %run a list of notebooks in Databricks

Question

I'd like to %run a list of notebooks from another Databricks notebook.

my_notebooks = ["./setup", "./do_the_main_thing", "./check_results"]
for notebook in my_notebooks:
   %run notebook

This doesn't work ofcourse.
I don't want to use dbutils.notebook.run() as this creates new jobs and doesn't return anything back - I want everything executable and queryable from the main notebook.

I thought perhaps it might be possible to import the actual module and run the function.

?%run shows the command points to IPython/core/magics/execution.py and run is a method of the class ExecutionMagics in the module execution. So perhaps, I could use execution.ExecutionMagic.run() if I created an instance of the class.

But it's beyond me - tricky and I'm doubting it's an effective solution.

How can this be done?

Am I really stuck with:-
%run ./a notebook

%run ./another_notebook

%run ./yet_another_hardcoded_notebook_name

Eternally grateful for any help!

Answer 1

Unfortunately it's not possible to do - %run doesn't allow to pass notebook name as a variable (see this answer with more details, and possible workaround).

Another approach would be to use so-called arbitrary files in repos functionality - if you define code as a Python file instead of notebook, then you'll be able to use it as normal Python module, and even load it dynamically if you need.