I want to run a notebook in databricks from another notebook using %run. Also I want to be able to send the path of the notebook that I'm running to the main notebook as a parameter.
The reason for not using dbutils.notebook.run is that I'm storing nested dictionaries in the notebook that's called and I wanna use them in the main notebook.
I'm looking for Something like:
path = "/References/parameterDefinition/schemaRepository"
%run <path variable>
Magic commands such as %run and %fs do not allow variables to be passed in.
The workaround is you can use dbutils as like dbutils.notebook.run(notebook, 300 ,{})
You can pass arguments as documented on Databricks web site: https://docs.databricks.com/notebooks/widgets.html#use-widgets-with-run
In the top notebook you can call
%run /path/to/notebook $X="10" $Y="1"
And then in the sub notebook, you can reference those arguments using the widgets API as in
x_value = dbutils.widgets.get("X")
y_value = dbutils.widgets.get("Y")
To your specific question, it would look something like this where "path" is the variable to be referenced via the widgets API in the target notebook:
%run /path/to/notebook $path="/path/to/notebook"
Unfortunately it's impossible to pass the path in %run as variable. You can pass variable as parameter only, and it's possible only in combination with with widgets - you can see the example in this answer. In this case you can have all your definitions in one notebook, and depending on the passed variable you can redefine the dictionary.
There will be a new functionality coming in the next months (approximately, see public roadmap webinar for more details) that will allow to import notebooks as libraries using the import statement. Potentially you can emulate the same functionality by exporting the notebook into the file on disk using the Export command of Workspace API, decoding the data & importing file's content, for example, if you have notebook called module1 with content
my_cool_dict = {"key1": "abc", "key2": 123}
then you can import it as following:
import requests
import base64
import os
api_url = dbutils.notebook.entry_point.getDbutils().notebook().getContext().apiUrl().get()
host_token = "your_PAT_token"
path = "/Users/..../module1"
# fetch notebook
response = requests.get(f"{api_url}/api/2.0/workspace/export",
json = {"format": "SOURCE", "path": path},
headers={"Authorization": f"Bearer {host_token}"}
).json()
# decode base64 encoded content
data = base64.b64decode(response["content"].encode("ascii"))
# write the file & __init__.py, so directory will considered a module
dir = os.path.join("/tmp","my_modules")
if not os.path.exists(dir):
os.mkdir(dir)
with open(os.path.join(dir, os.path.split(path)[-1]+".py"), "wb") as f:
f.write(data)
with open(os.path.join(dir, "__init__.py"), "wb") as f:
f.write("\n".encode("ascii"))
# add our directory into system path
import sys
sys.path.append(dir)
# import notebook
from module1 import my_cool_dict
and see that we got our variable:
You can't pass it as a variable while running the notebook like this:
In notebook1:
path_var = "/some/path"
%run ./notebook2
%path=path_var
However what you can do, and what I did, is access the dbutils object or the variable of notebook1 in notebook2:
In notebook1:
dbutils.widgets.text("path","", "")
path_var = "/some/path"
%run ./notebook2
Then in notebook2:
"""
No need to define path widget in Notebook2 like this:
dbutils.widgets.text("path","", "")
"""
path = dbutils.widgets.get("path")
print(path)
Output: /some/path
"""
Or you can access the path_var of Notebook1 directly
without defining it anywhere in Notebook2 like this
"""
print(path_var)
Output: /some/path
So this helps when you are using complicated variables such as heavily nested dictionaries8 in the notebook.
What I love about this approach is that environment of notebooks get shared when you call a notebook, meaning you can access variables & methods of Notebook1 in some Notebookn and vice versa such that:
