Calling a Google cloud function within Airflow DAG

Question

I have a google cloud function that is working, I am trying to call it from an Airflow DAG.

what I have tried so far is to use the SimpleHttpOperator:

MY_TASK_NAME = SimpleHttpOperator(
        task_id= "MY_TASK_NAME",
        method='POST',
        http_conn_id='http_default',
        endpoint='https://us-central1-myprojectname.cloudfunctions.net/MyFunctionName',
        data=({"schema": schema, "table": table}),
        headers={"Content-Type": "application/json"},
        xcom_push=False
    )

but digging into the logs, it says it cannot find the resource:

{base_task_runner.py:98} INFO - Subtask:
The requested URL /https://us-central1-myprojectname.cloudfunctions.net/MyFunctionName was not found on this server. That’s all we know.

also I noticed that it actually posts to https://www.google.com/ + the url I gave:

Sending 'POST' to url: https://www.google.com/https://us-central1-myprojectname.cloudfunctions.net/MyFunctionName

what is the proper way to call the function ? Thanks

https://stackoverflow.com/a/70174703/10375049 – Marco Cerliani Nov 30 '21 at 19:03 — Marco Cerliani, Nov 30 '21 at 19:03

score 2 · Accepted Answer · edited Sep 10 '21 at 22:08

2

This is because you are using the http_conn_id='http_default'.

The http_default connection looks as follows:

If you check the Hosts field, it says http://www.google.com/.

Either create a new Connection with HTTP Connection type or modify the http_default connection and change the host to https://us-central1-myprojectname.cloudfunctions.net/

Then update the endpoint field in your task to:

MY_TASK_NAME = SimpleHttpOperator(
        task_id= "MY_TASK_NAME",
        method='POST',
        http_conn_id='http_default',
        endpoint='MyFunctionName',
        data=({"schema": schema, "table": table}),
        headers={"Content-Type": "application/json"},
        xcom_push=False
    )

Edit: Added / at the end of URLs

edited Sep 10 '21 at 22:08

Sylvain Gantois

779
1
12
28

answered Aug 27 '19 at 09:40

kaxil

17,706
2
59
78

1

Thanks, it looks like I can call it now but I get a "ERROR - HTTP error: Forbidden", do you have an idea why it happens ? – Sylvain Gantois Aug 27 '19 at 21:46
Btw, it is a little detail but can you add a / at the end of the connection url, it needs one. – Sylvain Gantois Aug 27 '19 at 21:48
Yes you should be able to do that – kaxil Aug 28 '19 at 08:10
@SylvainGantois Hi , Did this work for you? – Snehil Singh Sep 09 '21 at 08:30
1

@SnehilSingh, it did but it was 2 years ago, I cannot guarantee it's still working. – Sylvain Gantois Sep 10 '21 at 05:27
1

@SnehilSingh After fixing the URL, I recall I had authentication issues, so you might want to check the other answer as well. – Sylvain Gantois Sep 10 '21 at 05:41

score 0 · Answer 2 · answered May 15 '21 at 11:28

As @kaxil noted you need to first change the http connection. You then need to be able to send the correct authenticating for invoking the cloud function. The below link has a step by step guide for doing this by subclassing SimpleHttpOperator

https://medium.com/google-cloud/calling-cloud-composer-to-cloud-functions-and-back-again-securely-8e65d783acce

As a side note, Google should make this process much clearer. It is perfectly reasonable to want to trigger a Google Cloud Function (gcf) from Google Cloud Composer. The documentation for how to send an http trigger to a gcf includes documentation for Cloud Scheduler, Cloud Tasks, Cloud Pub/Sub, and a host of others but not Cloud Composer

Calling a Google cloud function within Airflow DAG

2 Answers2