0

As far as I know, Google Big Query follows "eventual consistency" architecture, meaning that table creation, schema changes and data import is non-synchronous.

I'm building system that synchronize couple of tables with high update ratio to BQ (by schedule). Latter is not suited for updates at all, so I'm following idea of dropping all previous data and re-importing everything (data volume for this tables is low, so this seems feasible for now). I'm facing the issue with table re-creation: I'm deleting existing table, creating new (providing schema) and inserting data (via insert_rows, which does streamed insert, AFAIK).

And I'm experiencing inconsistency, most of the time (not always) ending up with empty table, but actual schema. From what I understand, this may happen in case of streaming insert still knows about old table, inserts data in it, and later on deletion/re-creation is synced up.

So my question is how to reliably get the point when table/schema updates is converged to all nodes/regions/etc? Or if there's any other way to reliably re-import the data with potential schema change.

Slam
  • 8,112
  • 1
  • 36
  • 44
  • Why do you need to delete the table and stream the data again? Why not use `CREATE OR REPLACE TABLE ... AS SELECT ...`? – Elliott Brossard Aug 23 '18 at 14:22
  • Does replacing somehow bypass "eventuality"? – Slam Aug 23 '18 at 14:29
  • When you actually delete the table and then recreate it, you end up with a new table identifier on the backend, and it takes ~10 minutes before streaming inserts will go to the new table. If you do an "in-place" replacement then you will have the same table ID. It's the same as if you use the `WRITE_TRUNCATE` option in a request. – Elliott Brossard Aug 23 '18 at 14:32

0 Answers0