0

In a local python file, I am using the databricks RunsApi to call a notebook in databricks and then return the response as a Json. When I run the notebook in databricks directly, it only takes about 3 seconds to run, but using the api is taking 25 seconds. Is there a way I can make this runtime shorter? This is the code I am using. Thanks

import os
import json
import time
from databricks_cli.sdk.api_client import ApiClient
from databricks_cli.runs.api import RunsApi

api_client = ApiClient(host=os.getenv('DATABRICKS_HOST'), token=os.getenv('DATABRICKS_TOKEN'))

runJson = {
            "name ": "api_test",
            "max_concurrent_runs" : 1,
            "tasks" : [
                {
                "task_key" : "api_test",
                "description" : "test",
                "notebook_task" :
                    {           
                    "notebook_path" : "path"
                    },
                "existing_cluster_id" : "id",
                "timeout_seconds" : 3600,
                "max_retries" : 3,
                "retry_on_timeout" : True
                }
            ]
            }

runs_api = RunsApi(api_client)
run_id = runs_api.submit_run(runJson)
runOutput = runs_api.get_run_output(run_id['run_id'])
J. Doe
  • 165
  • 5
  • 16
  • 1
    Hi, Thank you for asking a set of questions on this topic, we are trying to implement something very similar. Were you able to speed this up? Can you please share what worked for you? – rwg05 Aug 22 '23 at 23:47

0 Answers0