0

I have a database in cloud SQL Server. I need to move the database to BigQuery. Currently I'm using airflow to move one table at a time (downloading as .csv to gcs storage and then moving it to BQ).

Is there any other way to move all the tables in one go?

marc_s
  • 732,580
  • 175
  • 1,330
  • 1,459
Girish venkata
  • 704
  • 1
  • 7
  • 15
  • does this help? https://stackoverflow.com/questions/47624181/best-way-to-import-google-cloud-sql-data-into-bigquery https://www.google.com/search?q=google+cloud+sql+to+bigquery+migration&rlz=1C5GCEM_enUS1061US1062&oq=google+cloud+sql+to+bigquery+migration&aqs=chrome..69i57j69i64.7967j0j9&sourceid=chrome&ie=UTF-8 – Nicholas Hansen-Feruch Jul 10 '23 at 14:05
  • The problem is federated queries works only for cloud SQL postgres or MySQL db, not for SQL server – Girish venkata Jul 11 '23 at 21:25

1 Answers1

0

Another solution can be enabling CDC and continuously replicating data from SQL Server to BQ.

create and deploy a job that continuously replicates changed data from a Microsoft SQL Server database to a BigQuery table.

Or you can try @Nicholas Hansen-Feruch's recommendation of using federated query solution.

Nestor Ceniza Jr
  • 976
  • 3
  • 11
  • Federated queries is not working for cloud SQL SQL server, but I'll try CDC ( I know nothing about it) – Girish venkata Jul 11 '23 at 21:28
  • Also CDC won't work as I'm restoring the backup we receive as a new DB everytime (the data in cloud SQL SQL server is from backup file, delete the old db and restore the new data from the backup) – Girish venkata Jul 18 '23 at 16:55