2

We're launching a replication instance to replicate data from Mysql to BigQuery. After some hours, the instance is in Killed state.

When watching the logs, we have that (I added the 3 comments):

// DATA IS CORRECTLY LOADED IN BIGQUERY
2022-08-30 03:25:14,799 - INFO  [bq-daemon-2125:i.c.d.b.BigQueryEventConsumer@705] - Loading batch 1661829830877 of 933 events into staging table for [db.table]
2022-08-30 03:26:44,800 - INFO  [bq-daemon-2127:i.c.d.b.BigQueryEventConsumer@705] - Loading batch 1661829974724 of 1 events into staging table for [db.table]
2022-08-30 03:26:44,803 - INFO  [bq-daemon-2126:i.c.d.b.BigQueryEventConsumer@705] - Loading batch 1661829922476 of 914 events into staging table for [db.table]

// ERROR IS FROM HERE
2022-08-30 03:27:10,766 - INFO  [TwillContainerService:i.c.c.i.a.r.d.AbstractProgramTwillRunnable@281] - Program run program_run:[name].-SNAPSHOT.worker.DeltaWorker.[runid] completed. Releasing resources.
2022-08-30 03:27:10,768 - DEBUG [TwillContainerService:i.c.c.l.a.LogAppenderInitializer@137] - Stopping log appender TMSLogAppender
2022-08-30 03:27:42,635 - DEBUG [main:i.c.c.l.a.LogAppenderInitializer@137] - Stopping log appender TMSLogAppender
2022-08-30 03:27:46,410 - DEBUG [provisioning-task-2:i.c.c.i.p.t.ProvisioningTask@125] - Executing DEPROVISION subtask REQUESTING_DELETE for program run program_run:default.[name].-SNAPSHOT.worker.DeltaWorker.[runid].
2022-08-30 03:27:47,263 - DEBUG [provisioning-task-2:i.c.c.i.p.t.ProvisioningTask@129] - Completed DEPROVISION subtask REQUESTING_DELETE for program run program_run:default.[name].-SNAPSHOT.worker.DeltaWorker.[runid].
2022-08-30 03:28:17,819 - DEBUG [provisioning-task-2:i.c.c.i.p.t.ProvisioningTask@125] - Executing DEPROVISION subtask POLLING_DELETE for program run program_run:default.[name].-SNAPSHOT.worker.DeltaWorker.[runid].
2022-08-30 03:28:18,471 - DEBUG [provisioning-task-2:i.c.c.i.p.t.ProvisioningTask@129] - Completed DEPROVISION subtask POLLING_DELETE for program run program_run:default.[name].-SNAPSHOT.worker.DeltaWorker.[runid].
2022-08-30 03:28:21,230 - DEBUG [provisioning-task-2:i.c.c.i.p.t.ProvisioningTask@116] - Completed DEPROVISION task for program run program_run:default.[name].-SNAPSHOT.worker.DeltaWorker.[runid].

// THAT'S THE END OF THE LOGS, NOTHING ELSE

As you can see, data are correctly loaded in batchs in BigQuery, then suddenly, with no error message, the instance is deprovisioned.

It started about 3 weeks ago and since then, we constantly have to restart manually the worker.

Dagang
  • 24,586
  • 26
  • 88
  • 133
bjovanov
  • 471
  • 4
  • 13
  • What log is this? What does "replication instance" mean in this context? – Dagang Aug 31 '22 at 05:37
  • It's the log from the Data Fusion job that i downloaded. "Replication instance" means a replication job that we launched in Data Fusion and that we configured with a MySQL source and a BigQuery target so our data from our database are replicated in BigQuery. And this replication job is running inside a Data Fusion instance. – bjovanov Aug 31 '22 at 06:30
  • I see, I am not familiar with Data Fusion, not sure if it creates a Dataproc cluster or not in this case. – Dagang Aug 31 '22 at 07:30
  • @bjovanov This problem may occur if the schema gets changed after you enable the CDC, as DDL changes are not supported when CDC is enabled. If you have a support plan, I would recommend you to raise a ticket with [Google Cloud Support](https://cloud.google.com/support-hub). Or you can also create a thread in Google’s [Public Issue Tracker](https://cloud.google.com/support/docs/issue-trackers). – Sourav Dutta Sep 01 '22 at 09:11
  • It looks like something is telling the worker to stop. There may be useful information in the app-fabric logs (available from the System Admin page). If you search for the runid, it may give some clue about what triggered the stop. – Albert Shau Sep 06 '22 at 20:53

0 Answers0