0

I am trying to create a new Azure Databricks cluster using the Databricks Python SDK and I run into the following problem:

I need an Init Script for my cluster and when I am trying to set this property in code, I get the following error:

TypeError: 'InitScriptInfo' object is not iterable

For the cluster log conf property, same type of code works perfectly.

Create code is below. Do you have any ideas about how can I setup the InitScript in Pyhton SDK?

from databricks.sdk import WorkspaceClient
from databricks.sdk.service.compute import AutoScale, ClusterSource, DataSecurityMode,RuntimeEngine, ClusterLogConf, DbfsStorageInfo,InitScriptInfo, WorkspaceStorageInfo
w = WorkspaceClient(
  host  = workspace_host,
  token = workspace_token
)
c = w.clusters.create_and_wait(
  cluster_name              = 'TEST',
  spark_version             = '12.2.x-scala2.12',
  node_type_id              = 'Standard_D4s_v3',
  autotermination_minutes   = 15,
  num_workers               = 2,
  autoscale                 = AutoScale(min_workers=2, max_workers=4),
  cluster_log_conf          = ClusterLogConf(DbfsStorageInfo('dbfs:/cluster-logs')),
  init_scripts              = InitScriptInfo(WorkspaceStorageInfo('/Shared/filename.sh'))
)

Thank you

Alex Ott
  • 80,552
  • 8
  • 87
  • 132
AndreiGosman
  • 33
  • 4
  • As per the [link](https://databricks-sdk-py.readthedocs.io/en/latest/workspace/clusters.html) the init scripts option accepts a list of InitScriptInfo objects - `init_scripts – List[InitScriptInfo] (optional)` – rainingdistros Aug 21 '23 at 11:17

1 Answers1

1

When I tried to replicate the issue with your code in my environment, I got the same error:

enter image description here

The error you're getting, 'InitScriptInfo' object is not iterable, indicates that you are trying to iterate over an object that is not iterable, which means the object does not support iteration using a loop. I modified the code as mentioned below:

 init_scripts=[InitScriptInfo(WorkspaceStorageInfo('Filepath'))]

I tried to execute the code it executed successfully without any error:

enter image description here

The cluster created successfully.

enter image description here

Bhavani
  • 1,725
  • 1
  • 3
  • 6
  • Thank you for help, it is working. Just a small thing. Is it possible to indicate that init scripts source it is the Workspace? Seems that by default it goes to DBFS, whatever path I am writing in code. Thank you – AndreiGosman Aug 21 '23 at 13:06
  • Yes, you can do it. – Bhavani Aug 21 '23 at 13:14
  • Do you know how? I was expecting that WorkspaceStorageInfo to set directly a workspace type init script. Am I missing something here? Than kyou – AndreiGosman Aug 21 '23 at 13:24
  • I've figgure it out. I need to put all other storage types (S3, DBFS) on None and then it will create it with Workspace type. Like: [InitScriptInfo(S3StorageInfo(None), DbfsStorageInfo(None),WorkspaceStorageInfo('/Shared/filename.sh'))] – AndreiGosman Aug 21 '23 at 13:44