How to schedule Google Data Fusion pipeline?

Question

I have deployed a simple Data Fusion pipeline that reads from GCS and writes to BigQuery table.

I am looking for way to schedule the pipeline but could not find relevant documents.

Can anyone point me to documentation/pages that briefs about scheduling Data fusion pipelines?

score 5 · Accepted Answer · answered Apr 17 '19 at 16:33

5

You can schedule pipeline after deployment by clicking on Schedule button in the pipeline detail page. Once you click on it, you can configure the pipeline to run periodically.

Please see screenshots below:

answered Apr 17 '19 at 16:33

Sree

714
4
8

Is it possible to schedule the pipeline using external schedulers such as - Airlfow/Composer or any? – Jaison Apr 19 '19 at 04:56
Data Fusion uses REST apis for lifecycle management. It is currently not exposed to be used from external services. Plan is to expose it in Q2 2019, after which it will be possible to trigger the pipelines from external schedulers or airflow/composer. – Sree Apr 20 '19 at 16:52

score 5 · Answer 2 · answered Apr 18 '19 at 17:54

I was using "Data Fusion Basic Edition" which doesn't support scheduling and hence I was not able to find an option to schedule.

In Enterprise edition, I see an option "Schedule" after deploying the pipeline.

Feature comparisons here - Comparison between Basic and Enterprise edition

How to schedule Google Data Fusion pipeline?

2 Answers2

Linked