0

Could anyone help me in providing some pointers on how do we implement Data lineage on a DW type solution built on Google BigQuery using Google Cloud storage as source and Google Cloud Composer as the workflow manager to implement a series of SQL's?

navi
  • 11
  • 1

1 Answers1

0

If you have your data in Cloud Storage, you would maybe like to use something like GoogleCloudStorageToBigQueryOperator to first load your data in Bigquery, then use BigQueryOperator to run your queries.

Then you could see how your different DAGs,tasks etc are running in the Airflow Web UI inside Composer.

VictorGGl
  • 1,848
  • 10
  • 15