GCP Cloud Composer: get_client_id.py error with required arguments

Question

I have a question about GCP Cloud Composer.

To verify the function that triggers DAG (workflow) I would like to get the client ID by referring to the python code in the following article. https://github.com/GoogleCloudPlatform/python-docs-samples/blob/master/composer/rest/get_client_id.py

I get an error when I run the program.

The error that is appearing is as follows.

usage: id.py [-h] project_id location composer_environment
id.py: error: the following arguments are required: project_id, location, composer_environment

The meaning of the error is that the arguments project_id, location, composer_environment are not enough. I understand the meaning, but I am in trouble because the same error occurs no matter how I pass the arguments.

Below is a list of the commands and codes I have tried.


python3 id.py --project_id project_id --location location --composer_environment composer_environment

python3 id.py --project_id 'project_id' --location 'location' --composer_environment 'composer_environment'

python3 id.py --project_id --location --composer_environment

"""Get the client ID associated with a Cloud Composer environment."""

import argparse


def get_client_id(project_id, location, composer_environment):
    # [START composer_get_environment_client_id]
    import google.auth
    import google.auth.transport.requests
    import requests
    import six.moves.urllib.parse

    # Authenticate with Google Cloud.
    # See: https://cloud.google.com/docs/authentication/getting-started
    credentials, _ = google.auth.default(
        scopes=['https://www.googleapis.com/auth/cloud-platform'])
    authed_session = google.auth.transport.requests.AuthorizedSession(
        credentials)

    # project_id = 'YOUR_PROJECT_ID'
    # location = 'us-central1'
    # composer_environment = 'YOUR_COMPOSER_ENVIRONMENT_NAME'

    environment_url = (
        'https://composer.googleapis.com/v1beta1/projects/{}/locations/{}'
        '/environments/{}').format(project_id, location, composer_environment)
    composer_response = authed_session.request('GET', environment_url)
    environment_data = composer_response.json()
    airflow_uri = environment_data['config']['airflowUri']

    # The Composer environment response does not include the IAP client ID.
    # Make a second, unauthenticated HTTP request to the web server to get the
    # redirect URI.
    redirect_response = requests.get(airflow_uri, allow_redirects=False)
    redirect_location = redirect_response.headers['location']

    # Extract the client_id query parameter from the redirect.
    parsed = six.moves.urllib.parse.urlparse(redirect_location)
    query_string = six.moves.urllib.parse.parse_qs(parsed.query)
    print(query_string['client_id'][0])
    # [END composer_get_environment_client_id]


if __name__ == '__main__':
    parser = argparse.ArgumentParser(
        description=__doc__,
        formatter_class=argparse.RawDescriptionHelpFormatter)
    parser.add_argument('project_id', help='Your Project ID.')
    parser.add_argument(
        'location', help='Region of the Cloud Composer environent.')
    parser.add_argument(
        'composer_environment', help='Name of the Cloud Composer environent.')

    args = parser.parse_args()
    get_client_id(
        args.project_id, args.location, args.composer_environment)

An update for future folks looking at this - if you are using Composer 2, this script isn't compatible! (The underlying architecture is different, and you don't need the client ID for API calls) - please check out the linked code sample for the most recent version — LEC, Nov 05 '21 at 20:05

score 1 · Accepted Answer · answered Mar 01 '21 at 08:22

Add "--" prior to the set arguments. Prior to running script, make sure that are authenticated in your airflow webserver. Else you will have authentication errors when running the script.

    parser.add_argument('--project_id', help='Your Project ID.')
    parser.add_argument(
        '--location', help='Region of the Cloud Composer environent.')
    parser.add_argument(
        '--composer_environment', help='Name of the Cloud Composer environment.')

Sample run command:

python test.py --project_id=you-project-id-here --location=us-central1 --composer_environment=test-composer

Output:

xxxxx-xxxxxxx.apps.googleusercontent.com

score 1 · Answer 2 · answered Mar 19 '21 at 18:00

I'm the caretaker of this script right now and I'm so glad you brought this up and sorry this was not at all intuitive. There are a few options. The answer that suggests modifying the script and adding "--" prior to the arguments works (thanks Ricco D!). Additionally, you can pass the values themselves without the arguments at all. For example:

python get_client_id.py my-gcp-project us-west1 my-composer-environment-name

Either way though, I 100% agree that this isn't intuitive, and acknowledge that the error messaging that happens in this script is not the most helpful, and I'll work with my colleagues to give this script an update that improves the developer experience, has better error messaging, and is in line with GCP DevRel practices.

GCP Cloud Composer: get_client_id.py error with required arguments

2 Answers2