I have built a BigQuery data lineage system using the document provided by Google. https://cloud.google.com/architecture/building-a-bigquery-data-lineage-solution
I was able to generate the table lineage for all the SQLs running on bigquery in my project. But when it comes to column lineage, the lineage is captured only for SELECT statements.
I need to capture the column lineage for StoredProcedures, CREATE OR REPLACE, etc.
Is there a solution ?
Code from Google: https://github.com/GoogleCloudPlatform/bigquery-data-lineage.git
Updated code from my java resource: https://github.com/ivktiwari/bigquery-lineage/blob/main/test_mod.zip
The updated code has resolved all the exceptions and made the dataflow job capture lineage.
The only thing remaining is the ColumnLineage for all the SQL queries triggered.
