0

I want to export the data from Avro files which are present in GCS into BigQuery table using Dataflow and Python. Can anyone let me know how to do it? Because in dataflow there is no readymade template for Transferring Avro files in Batch mode to BigQuery table.

Rahul Wagh
  • 281
  • 6
  • 20
  • So you already have a template. Go ahead use it if you dont want any transformations to be applied to the data. – Mr.Batra Jan 05 '22 at 10:17
  • Mr.Batra, no as I said in Dataflow there isn't any readymade template available to transfer data from GCS Avro to BigQuery table. There is a template but using that you can transfer Streaming Avro data from dataflow to BigTable but not in BigQuery table. – Rahul Wagh Jan 05 '22 at 12:26
  • 2
    Any reason for the desire to use Dataflow? The BigQuery API already has support for easily loading avro files (see code: https://cloud.google.com/bigquery/docs/loading-data-cloud-storage-avro#python) and (bonus) it supports schema evolution which using Dataflow will not. – jamiet Jan 05 '22 at 22:46
  • The data is huge and I want to know how much it will cost if I use Dataflow. – Rahul Wagh Jan 06 '22 at 10:03

0 Answers0