0

I plan to have multiple App engine instances read/write from the same BigQuery table. What's the best way (memcache, datastore, taskqueue,etc.. ) to handle this? Each instance will read from a config table and process rows.

Is there a simple sample that I can follow?

Optionally, If I create one instance (with a back-up) and it is under load that Google spins up a second, will resource sharing be handled automatically or do I need to handle it programmatically?

DElig
  • 23
  • 3
  • Could you clarify your use case a bit more? You are worried that there will be issues with two instances of App Engine accessing the same BigQuery table? – A.Queue Apr 30 '18 at 09:37
  • Did you atleast check [Introduction to Optimizing Query Performance](https://cloud.google.com/bigquery/docs/best-practices-performance-overview) for starters? – ReyAnthonyRenacia Apr 30 '18 at 11:31

1 Answers1

0

In case you want to do many small reads and writes to BigQuery then you probably will be better off with CloudSQL as said here:

Using point-specific DML statements is an attempt to treat BigQuery like an Online Transaction Processing (OLTP) system. BigQuery focuses on Online Analytical Processing (OLAP) by using table scans and not point lookups. If you need OLTP-like behavior (single-row updates or inserts), consider a database designed to support OLTP use cases such as Google Cloud SQL.

You can also handle small data which is updated very often by using Cloud Storage as an external data source for your queries. For example, you can use it if you have 10 megabytes of data that is being constantly updated and 2 TB that are updated rarely.

Read more on optimization and find your example with App Engine here.

A.Queue
  • 1,549
  • 6
  • 21