How to send a list as parameter in databricks notebook task?

Question

I am using Databricks Resi API to create a job with notebook_task in an existing cluster and getting the job_id in return. Then I am calling the run-now api to trigger the job. In this step, I want to send a list as argument via the notebook_params, which throws an error saying "Expected non-array for field value".

Is there any way I can send a list as an argument to the job?

I have tried sending the list argument in base_params as well with same error.

user_json={
                                "name": job_name,
                                "existing_cluster_id": cluster_id,
                                "notebook_task": {
                                                        "notebook_path": notebook_path
                                                    },
                                "email_notifications":{
                                "on_failure":[email_id]
                                },
                                "max_retries": 0,
                                "timeout_seconds": 3600
                                                }

response=requests.post('https://<databricks_uri>/2.0/jobs/create',headers=head,json=user_json,timeout=5, verify=False)

job_id=response.json()['job_id']

json_job={"job_id":job_id,"notebook_params":{"name":"john doe","my_list":my_list}}

response = requests.post('https://<databricks_uri>/2.0/jobs/run-now', headers=head, json=json_job, timeout=200, verify=False)

score 3 · Answer 1 · answered Aug 16 '19 at 10:31

Not found any native solution yet, but my solution was to pass the list as a string and parse it back out on the other side:

json_job={"job_id":job_id,
          "notebook_params":{
                "name":"john doe",
                "my_list":"spam,eggs"
           }
}

Then in databricks:

my_list=dbutils.widgets.get("my_list")
my_list=my_list.split(",")

With appropriate care around special characters or e.g. conversion to numeric types.

If the objects in the list are more substantial, then sending them as a file to dbfs using the CLI or API before running the job may be another option to explore.

score 0 · Answer 2 · answered Dec 21 '22 at 08:55

Hi may be I'm bit late but found a better solution.

Step 1: Use JSON.stringyfy() in the console of any browser to convert your value(object, array, JSON etc) into string

Ex:

Now use this value in the body of URL

In Databricks notebook convert string to JSON using python json module.

Hope this helps

How to send a list as parameter in databricks notebook task?

2 Answers2