1

So I'm trying to provision a Google Dataproc cluster in a Python API. Below is the function I use for creating a cluster, but when I try running the script, it spits out an error, so something is wrong, but I can't tell what.

    def create_cluster(dataproc, bucket, project, zone, region, cluster_name, master_type, worker_type, image_version, init):
        print('Creating cluster...')
        zone_uri = 'https://www.googleapis.com/compute/v1/projects/{}/zones/{}'.format(
        project, zone)
        cluster_data = {
            'projectId': project,
            'clusterName': cluster_name,
            'config': {
                'configBucket': bucket,
                'gceClusterConfig': {
                    'zoneUri': zone_uri,
                    'metadata': { 
                        'HASH': '6e815ac3d973', 
                        'SPARK': '2.0.2' , 
                        'HAIL_VERSION': '0.1' 
                    }
                },
                'masterConfig': { 
                    'machineTypeUri' : master_type
                },
                'workerConfig': {
                    'machineTypeUri' : worker_type
                },
                'softwareConfig': {
                    'imageVersion': image_version
                },
                'initializationActions': [  
                  {
                    'executableFile': 'gs://{}/{}'.format(bucket, init) 
                  }
                 ]
               }
            }


         result = dataproc.projects().regions().clusters().create(
            projectId=project,
            region=region,
            body=cluster_data).execute()
         return result

It works when I create a cluster via: gcloud dataproc clusters create testing-automation --image-version 1.1 --project perfect-atrium-179917 --bucket hail-qc-data --master-machine-type n1-standard-2 --worker-machine-type n1-standard-2 --zone us-east4-b --initialization-actions gs://hail-qc-data/initialization_action.sh --metadata HASH=6e815ac3d973,SPARK=2.0.2,HAIL_VERSION=0.1

claudiadast
  • 419
  • 1
  • 9
  • 18

0 Answers0