3

Google cloud function cannot be used to trigger the composer DAG on Pub/Sub message

I have tried the PubSubPullSensor 

pull_messages = PubSubPullSensor(
    task_id="pull_messages",
    ack_messages=True,
    project='xxxx',
    subscription='xxxx',
)

but this doesn't trigger ,as DAG as expected

Any help to trigger the DAG on Pub/Sub message will be greatly appreciated?

manish dev
  • 97
  • 1
  • 9
  • By "Google cloud function cannot be used to trigger the composer DAG on Pub/Sub message" you meant that one of your system requirements is to not use Google cloud Functions? – Tlaquetzal Feb 08 '20 at 16:04
  • @Tlaquetzal yes you are correct , that my system requirement is such that cloud funtions cannot be used but also the DAG has to be triggered on a Pub/Sub message – manish dev Feb 08 '20 at 16:14

1 Answers1

4

The PubSubPullSensor is part of a DAG. The dag needs to be running in order to the sensor to be executed, that's why it didn't work.

I think that the most striaghtforward approach would be to use cloud functions, but if that's not possible, the second option would be to use another server (it could be another computing option within GCP: Cloud AppEngine, Cloud Run, etc.) to receive the Pub/Sub message and trigger the dag. Basically, the same idea as Pub/Sub + Cloud Functions, without functions.

If you don't want to follow this approach, and want to trigger the Dag within the same Composer environment, you could use PubSubPullSensor on a running dag and use the TriggerDagRunOperator when needed.

This idea can be used in many forms; however, the tricky thing is that PubSubPullSensor needs to be executed in a running dag. A possible solution for this, is to schedule the dag to run often, for example every 5 minutes

Tlaquetzal
  • 2,760
  • 1
  • 12
  • 18
  • Thanks. I need to trigger the DAG on a pub/sub message notification, I would try the approach with another server – manish dev Feb 08 '20 at 18:16
  • [Here's](https://cloud.google.com/composer/docs/how-to/using/triggering-with-gcf) an example using cloud functions. You can take the core of the code to use it elsewhere. – Tlaquetzal Feb 08 '20 at 18:28
  • Thanks @Tlaquetzal. I am following the approach suggested by you that the DAG can be scheduled for 5 minutes and then the pub sub messages can be pulled and the next task in the DAG can be successfully executed – manish dev Feb 08 '20 at 21:29