1

I am trying to implement a BI solution using GCP where I have data in flat files in cloud datastore and I have to push this data in my Data Warehouse on BigQuery. The data will be incremental after the first load.

There doesn't seem to be any ETL functionality which I can use to implement this incremental data load into my warehouse. Using Cloud Dataflow, I can push the delta load into the BigQuery tables but this approach doesn't handle the updated records correctly.

Can anyone suggest here what could be the best approach for implementing this solution?

Abhishek Mittal
  • 21
  • 5
  • 1
    please see this post https://stackoverflow.com/a/55673711/1031958 and let us know if it answers your question. If not please provide a more detail problem you are facing and what you tried so far which is not working – Tamir Klein Apr 15 '19 at 11:18
  • You can try airflow schedule push your incremental data into BigQuery table – howie Apr 15 '19 at 13:26
  • Yes, airflow seems to be an option but that requires a good coding knowledge while I'm trying to figure out an approach which can be implemented easily by a traditional ETL developers. – Abhishek Mittal Apr 16 '19 at 05:36

0 Answers0