We're launching a replication instance to replicate data from Mysql to BigQuery. After some hours, the instance is in Killed
state.
When watching the logs, we have that (I added the 3 comments):
// DATA IS CORRECTLY LOADED IN BIGQUERY
2022-08-30 03:25:14,799 - INFO [bq-daemon-2125:i.c.d.b.BigQueryEventConsumer@705] - Loading batch 1661829830877 of 933 events into staging table for [db.table]
2022-08-30 03:26:44,800 - INFO [bq-daemon-2127:i.c.d.b.BigQueryEventConsumer@705] - Loading batch 1661829974724 of 1 events into staging table for [db.table]
2022-08-30 03:26:44,803 - INFO [bq-daemon-2126:i.c.d.b.BigQueryEventConsumer@705] - Loading batch 1661829922476 of 914 events into staging table for [db.table]
// ERROR IS FROM HERE
2022-08-30 03:27:10,766 - INFO [TwillContainerService:i.c.c.i.a.r.d.AbstractProgramTwillRunnable@281] - Program run program_run:[name].-SNAPSHOT.worker.DeltaWorker.[runid] completed. Releasing resources.
2022-08-30 03:27:10,768 - DEBUG [TwillContainerService:i.c.c.l.a.LogAppenderInitializer@137] - Stopping log appender TMSLogAppender
2022-08-30 03:27:42,635 - DEBUG [main:i.c.c.l.a.LogAppenderInitializer@137] - Stopping log appender TMSLogAppender
2022-08-30 03:27:46,410 - DEBUG [provisioning-task-2:i.c.c.i.p.t.ProvisioningTask@125] - Executing DEPROVISION subtask REQUESTING_DELETE for program run program_run:default.[name].-SNAPSHOT.worker.DeltaWorker.[runid].
2022-08-30 03:27:47,263 - DEBUG [provisioning-task-2:i.c.c.i.p.t.ProvisioningTask@129] - Completed DEPROVISION subtask REQUESTING_DELETE for program run program_run:default.[name].-SNAPSHOT.worker.DeltaWorker.[runid].
2022-08-30 03:28:17,819 - DEBUG [provisioning-task-2:i.c.c.i.p.t.ProvisioningTask@125] - Executing DEPROVISION subtask POLLING_DELETE for program run program_run:default.[name].-SNAPSHOT.worker.DeltaWorker.[runid].
2022-08-30 03:28:18,471 - DEBUG [provisioning-task-2:i.c.c.i.p.t.ProvisioningTask@129] - Completed DEPROVISION subtask POLLING_DELETE for program run program_run:default.[name].-SNAPSHOT.worker.DeltaWorker.[runid].
2022-08-30 03:28:21,230 - DEBUG [provisioning-task-2:i.c.c.i.p.t.ProvisioningTask@116] - Completed DEPROVISION task for program run program_run:default.[name].-SNAPSHOT.worker.DeltaWorker.[runid].
// THAT'S THE END OF THE LOGS, NOTHING ELSE
As you can see, data are correctly loaded in batchs in BigQuery, then suddenly, with no error message, the instance is deprovisioned.
It started about 3 weeks ago and since then, we constantly have to restart manually the worker.