3

I am looking to connect to an external API through HTTPS with Airflow.

To do that, I configure my http Airflow connector regarding the documentation.

I setted my host with my url : myurl.com Then i setted the schema value to 'https' as expected in the documentation.

And because I need a certificate, according to the http_hook documentation (airflow doc) I added extra option : {"cert":"/home/airflow/gcs/data/mycertificate.pem"}

I am using Composer on the Google Cloud Platform, so I putted my certificate in the corresponding google cloud storage bucket (doc)

I am not sure about my configuration but when I try to use the SimpleHttpOperator

get_token = SimpleHttpOperator(
    task_id='get_access_token',
    method='POST',
    headers={
        "Authorization": "Basic mytooken=="},
    endpoint='/SASLogon/oauth/token',
    http_conn_id='myconnid',
    trigger_rule="all_done",
    response_filter=lambda response: response.json()['access_token']
)

get_token

I have the following error :

ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: unable to get local issuer certificate (_ssl.c:1131)

I am not sure if the error is due to my cert which is not found or due to my self-signed certificate.

Anyway, I am pretty sure I have got to add a configuration somewhere because my certificate is self-signed.

Have you got any idea ?

Baptiste Moisson
  • 33
  • 3

2 Answers2

1

The SimpleHttpOperator is just a simple call to Python's Requests library.

If you are googling this question, familiarize yourself with this stuff I guess:

mts
  • 166
  • 2
  • 9
0

Maybe is too late, anyway I resolved the same issue like that:

get_token = SimpleHttpOperator(
    task_id='get_access_token',
    method='POST',
    headers={
        "Authorization": "Basic mytooken=="},
    endpoint='/SASLogon/oauth/token',
    http_conn_id='myconnid',
    trigger_rule="all_done",
    response_filter=lambda response: response.json()['access_token'],
    extra_options={"verify":"/home/airflow/gcs/data/<you_pem>.pem"}
)

Anyway best way to solve that for me should be to have a complete Airflow connection, so installing the pem inside the Airflow Webserver and thus eliminating the need to specify extra_options at task level.

NOTE: In my case the pem was all the certificate chain, no just the final certificate.

Giorgio
  • 1,073
  • 3
  • 15
  • 33