1

I'm trying to setup a DAG which would respond to Cloud Pub/Sub messages. It needs me to add the following import statements in my DAG code:

from airflow.providers.google.cloud.operators.pubsub import (
PubSubCreateSubscriptionOperator, PubSubCreateTopicOperator, PubSubDeleteSubscriptionOperator,
PubSubDeleteTopicOperator, PubSubPublishMessageOperator,
)
from airflow.providers.google.cloud.sensors.pubsub import PubSubPullSensor

DAG import is failing as it is unable to resolve dependencies. Can anyone let me know the required dependencies and how to bring them within Cloud Composer environment?

Balajee Venkatesh
  • 1,041
  • 2
  • 18
  • 39

1 Answers1

5

At the moment, Google Cloud Composer has only released up to Airflow V1.10.3. The folder structure for V1.10.3 or V1.10.2 is different from the one in the current master branch. And the names of operators might also be different.

So the import for the Google Cloud Pub/Sub operators and sensor should be like this:

from airflow.contrib.operators.pubsub_operator import (
    PubSubPublishOperator, PubSubSubscriptionCreateOperator,
    PubSubSubscriptionDeleteOperator, PubSubTopicCreateOperator,
    PubSubTopicDeleteOperator)
from airflow.contrib.sensors.pubsub_sensor import PubSubPullSensor

Reference:

V1.10.2 Pub/Sub

https://github.com/apache/airflow/blob/1.10.2/airflow/contrib/operators/pubsub_operator.py https://github.com/apache/airflow/blob/1.10.2/airflow/contrib/sensors/pubsub_sensor.py

V1.10.3 Pub/Sub

https://github.com/apache/airflow/blob/1.10.3/airflow/contrib/operators/pubsub_operator.py https://github.com/apache/airflow/blob/1.10.3/airflow/contrib/sensors/pubsub_sensor.py

Ryan Yuan
  • 2,396
  • 2
  • 13
  • 23
  • 1
    It resolves the Pub/Sub integration with Airflow DAG. However, I would like to understand if there is any way to kick-off the DAG automatically whenever a new messages arrives on the Pub/Sub topic. One way is to setup a cloud function as a watcher of incoming messages. Is there any direct way in which Airflow itself can detect the incoming message on Pub/Sub topic and trigger itself as an action of event. – Balajee Venkatesh Jan 30 '20 at 05:27
  • @BalajeeVenkatesh Answering your question here: https://stackoverflow.com/a/58575463/9465561 – Ryan Yuan Jan 30 '20 at 05:44