Object embedded in Databricks SQL command

Question

I came across the following SQL command in Databricks notebook and I am confused about what is this ${da.paths.working_dir} object in following SQL command. Is it a python object or something else?

SELECT * FROM parquet.${da.paths.working_dir}/weather

I know it contains the path of a working directory but how can I access/print it. I tried to demystify it but failed as illustrated in the following figure.

NOTE: My notebook is SQL notebook

I even check the same code using Scala magic command but still got an error. In a later video, Instructor mentioned at one place, that this is a high-level variable. But still, I am confused this is a high-level variable in what (Python, SQL, Scala, bash)? — Muhammad Umar Amanat, May 24 '22 at 09:07

Muhammad Umar Amanat · Accepted Answer · 2022-06-21T11:25:02.287

Finally, I figured it out. This is a high-level variable in Databricks SQL and we can access it using the SELECT keyword in Databricks SQL as shown below:

SELECT '${da.paths.working_dir}';

EDIT: This high variable is spark configuration which can be set as follows:

## spark.conf.set(key, value)
spark.conf.set(da.paths.working_dir, "/path/to/files")

To access this property in python:

spark.conf.get(da.paths.working_dir)

To access this property in Databricks SQL:

SELECT {da.paths.working_dir}

Object embedded in Databricks SQL command

1 Answers1