0

I am setting up entire GCP architecture using Deployment Manager using Python template structure.

I have tried to execute the script below:

        'name': 'dataproccluster',
        'type': 'dataproc.py',
        'subnetwork': 'default',
        'properties': {
            'zone': ZONE_NORTH,
            'region': REGION_NORTH,
            'serviceAccountEmail': 'X@appspot.gserviceaccount.com',
            'softwareConfig': {
                'imageVersion': '1.4-debian9',
                'properties': {
                    'dataproc:dataproc.conscrypt.provider.enable' : 'False'
                }
            },
            'master': {
                'numInstances': 1,
                'machineType': 'n1-standard-1',
                'diskSizeGb': 50,
                'diskType': 'pd-standard',
                'numLocalSsds': 0
            }, 
            'worker': {
                'numInstances': 2,
                'machineType': 'n1-standard-1',
                'diskType': 'pd-standard',
                'diskSizeGb': 50,
                'numLocalSsds': 0
            },
            'initializationActions':[{
                'executableFile': 'gs://dataproc-initialization-actions/python/pip-install.sh'
            }],
            'metadata': {
                'PIP_PACKAGES':'requests_toolbelt==0.9.1 google-auth==1.6.31'
            },
            'labels': {
                'environment': 'dev',
                'data_type': 'X'
            }
        }

Which results in the following error:

Initialization action failed. Failed action 'gs://dataproc-initialization-actions/python/pip-install.sh',\

I would like to evaluate if it is an error on my side, or an API problem of any sort? I found Google tickets related to this topic covering CLI deployment, however they were marked as solved. I found nothing on Deployment Manager side.

If it is an error on my side what am I doing wrong?

Stefan G.
  • 890
  • 5
  • 10
Organic Platypus
  • 61
  • 7
  • 3
    I was able to get this working by changing the metadata to: 'PIP_PACKAGES':'requests_toolbelt==0.9.1 google-auth==1.6.3' I used googe-auth library version"1.6.3" and it [deployed](https://pypi.org/project/google-auth/) successfully. if this doesn't fix the issue, you can always route this issue to dataproc-initialization-actions [GitHub page](https://github.com/GoogleCloudPlatform/dataproc-initialization-actions/issues) and we will assist you. – Milad Tabrizi Oct 30 '19 at 16:35
  • This is an issue with the dataproc API not the DM API. If you take the same expanded manifest that DM creates and make an API call without DM, you will get the same error message – Patrick W Nov 01 '19 at 18:41

0 Answers0