1

I've created a Data Fusion replication job to replicate some tables on a test database.

It works well at the beginning if I don't change the tables schema. But I've added a new column and that column is ignored from the replication job. I guess that if I create a new table, even that table would be ignored.

Is there a way to include schema updates (new table, update column field, new column etc...) inside an already running Data Fusion replication job?

I guess a possible solution would be to stop the currently running job and create a new one including new tables, new columns etc... but I'd like to avoid that a new job would replicate all the database again.

Any possible solution?

marc_s
  • 732,580
  • 175
  • 1,330
  • 1,459
Alessio Innocenzi
  • 401
  • 5
  • 19

1 Answers1

0

Unfortunately, Data Fusion Replication for SQL server currently does not support DDL propagation during runtime; you will need to delete and recreate the replicator pipeline in order to propagate any changes to schema to the new BigQuery table.

One way to avoid replicating existing data with DDL change is that , you can manually modify the BigQuery table schema (But BigQuery also has limited support for schema changes) and create a new replication job and disable replicating existing data(there is an option that let you choose whether to replicate existing data, default is true)

kiran mathew
  • 1,882
  • 1
  • 3
  • 10
  • Thank you. I've updated my bigquery table by adding the new column manually and the replication job has worked well. Anyway, in order to make it work, sqlserver CDC must be updated with the newly added column, otherwise datafusion is not able to intercept the changes. I was struggling with this detail since it was my first time with CDC – Alessio Innocenzi Nov 10 '22 at 10:47
  • I am facing the same issue due to which some of the data was lost in BQ. – Ishan Kumar Dec 04 '22 at 21:29