1

I'm trying to automate creation of connector in airflow by github action, but since it is an external provider, the payload that need to be sent to airflow REST API doesn't work and i didn't find any documentation on how to do it.

So here is the PAYLOAD i'm trying to send :

PAYLOAD = {
    "connection_id": CONNECTOR,
    "conn_type": "google_cloud_platform",
    "extra": json.dumps({
        "google_cloud_platform": {
            "keyfile_dict" : open(CONNECTOR_SERVICE_ACCOUNT_FILE, "r").read(),
            "num_retries" : 2,
        }
    })
}

According to the airflow documentation here

And the information i found on the "create connector" page of airflow UI : Airflow UI create connector page

But i received no error (code 200) and the connector is created but doesn't have the settings i tried to configure.

I confirm the creation works on the UI.

Does anyone have a solution or document that refer to the exact right payload i need to sent to airflow rest api ? Or maybe i miss something.

  • Airflow version : 2.2.3+composer
  • Cloud Composer version (GCP) : 2.0.3
  • Github runner version : 2.288.1
  • Language : Python

Thanks guys and feel free to contact me for further questions.

Bye

2 Answers2

1

@vdolez was write, it's kind of a pain to format the payload to have the exact same format airflow REST API want. it's something like this :

"{\"extra__google_cloud_platform__key_path\": \"\", 
\"extra__google_cloud_platform__key_secret_name\": \"\", 
\"extra__google_cloud_platform__keyfile_dict\": \"{}\", 
\"extra__google_cloud_platform__num_retries\": 5, 
\"extra__google_cloud_platform__project\": \"\", 
\"extra__google_cloud_platform__scope\": \"\"}"

And when you need to nest dictionnary inside some of these field, not worth the time and effort. But in case someone want to know, you have to escape every special character.

I change my workflow to notify competent users to create connector manually after my pipeline succeed.

I will try to contact airflow/cloud composer support to see if we can have a feature for better formatting.

0

You might be running into encoding/decoding issues while sending data over the web.

Since you're using Composer, it might be a good idea to use Composer CLI to create a connection.

Here's how to run airflow commands in Composer:

gcloud composer environments run ENVIRONMENT_NAME \
    --location LOCATION \
    SUBCOMMAND \
    -- SUBCOMMAND_ARGUMENTS

Here's how to create a connection with the native Airflow commands:

airflow connections add 'my_prod_db' \
    --conn-type 'my-conn-type' \
    --conn-login 'login' \
    --conn-password 'password' \
    --conn-host 'host' \
    --conn-port 'port' \
    --conn-schema 'schema' \
    ...

Combining the two, you'll get something like:

gcloud composer environments run ENVIRONMENT_NAME \
    --location LOCATION \
    connections \
    -- add 'my_prod_db' \
    --conn-type 'my-conn-type' \
    --conn-login 'login' \
    --conn-password 'password' \
    --conn-host 'host' \
    --conn-port 'port' \
    --conn-schema 'schema' \
    ...

You could run this in a Docker image where gcloud is already installed.

vdolez
  • 977
  • 1
  • 14
  • 33
  • I want to do it from github actions, since i'm not using docker for the moment as an executor, i could ask for another solution but still yours are good. The problem is you are referencing the documentation for a basic airflow native connector, what about external connector settings like with the google provider in airflow ? – Philippe LETAIF Mar 18 '22 at 10:09
  • @PhilippeLETAIF you can actually use [Docker from GitHub Actions](https://docs.github.com/en/actions/creating-actions/creating-a-docker-container-action) and using the CLI is probably a safer implementation (for handling credentials etc.) If the connexion created from the UI is good, you could [export it to a file](https://airflow.apache.org/docs/apache-airflow/stable/howto/connection.html#exporting-connections-from-the-cli) and then [use that file to create the connexion](https://airflow.apache.org/docs/apache-airflow/stable/cli-and-env-variables-ref.html#import) using the CLI. – vdolez Mar 18 '22 at 10:44
  • @PhilippeLETAIF I added [this link](https://airflow.apache.org/docs/apache-airflow/stable/howto/connection.html#handling-of-special-characters-in-connection-params) to my answer in order to help you escape potential special chars. – vdolez Mar 18 '22 at 10:46
  • Yes, i understand, will try asap then and come back for feedback. I forgot to mention the composer v2 is hosted on a GKE that is autopilot and with private control plane, meaning i can not have access from external endpoint (policy), the cli will still work ? – Philippe LETAIF Mar 18 '22 at 11:02
  • @PhilippeLETAIF yeah, the CLI is designed to work internally. You should use a dedicated Service Account with the appropriate rights from GitHub in order to be able use the CLI (for example `roles/composer.admin` might be a bit overprivileged) – vdolez Mar 18 '22 at 13:22
  • 1
    Ok, i managed to export the connection to see how airflow format the connection for external provider. ALL is stored in extra but in a format that is not human readable. Something like a json that is inline stringified with all the quots escaped, here is an example : `"extra": "{\"extra__google_cloud_platform__key_path\": \"\", \"extra__google_cloud_platform__key_secret_name\": \"\"` It means that whatever i use (REST or CLI) the format encoding will be a pain, so i will continue using REST API for consistency with my worflow Thanks for all the clarification you have provided me ;) – Philippe LETAIF Mar 21 '22 at 09:28
  • @PhilippeLETAIF don't hesitate to post an answer once you're able to do it ! Good luck – vdolez Mar 21 '22 at 10:37