6

I keep hitting an issue where BigQuery's API tells me a table I just created doesn't exist. According to it's documentation on errors for streaming inserts

"Because BigQuery's streaming API is designed for high insertion rates, modifications to the underlying table metadata exhibit are eventually consistent when interacting with the streaming system."

https://cloud.google.com/bigquery/docs/error-messages#metadata-errors-for-streaming-inserts

However, Google also say "All table modifications in BigQuery, including DML operations, queries with destination tables, and load jobs are ACID-compliant" (emphasis is mine).

https://cloud.google.com/architecture/bigquery-data-warehouse#handling_change

So, if creation of the table is eventually consistent, is it BASE, or is it ACID and I am not thinking about it correctly as the eventual consistency is with the API, not within the DB.

James Dewes
  • 387
  • 4
  • 22
  • I have try to create a big query table and hit it using [streaming insert sample](https://cloud.google.com/bigquery/docs/samples/bigquery-table-insert-rows#code-sample) and found no issue whatsoever maybe I'm not sure on that part about your initial issue. On the other hand, I understand that whenever we operate bigquery (ie: ui, api calls) if the operation fits the ACID-compliant definition. it can be consider as ACID. BigQuery itself is BASE/ACID, [check this](https://stackoverflow.com/questions/28037551/does-google-bigquery-amazon-redshift-use-column-based-relational-database-or-no). – Betjens Oct 21 '21 at 12:55
  • So it would seem that it is specific to the streaming API and the eventual consistency of the metadata for that API. More info is here: https://cloud.google.com/bigquery/docs/error-messages#streaming. Looking back at the quote above streaming inserts are not included in the list of all modifications. – James Dewes Jan 21 '22 at 22:03

0 Answers0