5

We have a app that is hosted in US and the EU. For great performance we will host a datastore instance on both locations.

Now is our question how we can replicate the data from the US datastore to the EU datastore?

Or is it no matter where we store the data from technical view?

Dan McGrath
  • 41,220
  • 11
  • 99
  • 130
Aron
  • 1,179
  • 15
  • 29

1 Answers1

3

There is not such thing as multi-location GAE apps (at least not yet). The app's unique location can be selected when the app is created and cannot be changed. See App Engine - How to create project in region us-central and gae app moving to eu-datacenter or re-opening

An important note from Creating a project and application:

Important: Each Cloud Platform project can contain only a single App Engine application and you cannot change the region after you create the App Engine application.

Now the datastore itself can only exist in a single cloud project, which also maps 1:1 to a location. From Google Cloud Datastore Locations:

When you create a Google Cloud Platform project, you must choose a location where the project's data is stored. To reduce latency and increase availability, store your data close to the users and services that need it. You can create projects in the following locations:

  • europe-west — Western Europe
  • us-central — Central United States
  • us-east1 — Eastern United States
  • asia-northeast1 — Northeastern Asia-Pacific

Both App Engine and Cloud Datastore are listed as products with Multi-Regional Coverage (see https://cloud.google.com/about/locations/). But that coverage is not global, it still remains contained within a region group. And US and EU are in different region groups, so this won't help you. From Multi-regional resources:

The data associated with multi-regional resources is not tied to a specific region and can be moved between regions and regions can be added and removed from a region group. For example, buckets in the European Union location for Google Cloud Storage keep data at-rest inside the European Union, but at-rest data can be stored in or moved to any Cloud Storage region within the European Union (subject to terms of service and service specific terms).

So at best you can have 2 different GAE apps deployed in 2 different regions, eventually from (almost) the same code.

You need to take care of replicating the data across these apps yourself, for example by adding custom/dedicated communication between the apps for that purpose or by using the Remote API (each app acting like a client for the other app's remote API).

Most likely not trivial, especially if the data consistency is important. In such case you might be better off with a single app and some performance penalty for some of your customers, based on their location.

Community
  • 1
  • 1
Dan Cornilescu
  • 39,470
  • 12
  • 57
  • 97
  • Note: Cloud Datastore doesn't have used with only App Engine Standard. Using a different compute environment would all you to have your app deployed in manner regions. With local caching but accessing a central database in say us-central, you could get better performance at the cost of some cache consistency. – Dan McGrath Dec 28 '16 at 23:26
  • Thanks for the answer, what is the best solution when you work with customers from the EU and the new Data protection law? – Aron Dec 29 '16 at 09:58
  • I'd have 2 separate per-region apps, but rather independent from each-other - no data replication. If the app's functionality demands inter-site interaction I'd implement it in a different way. Replicating stored personal data from EU to US might actually be violating that law, for example. – Dan Cornilescu Dec 29 '16 at 14:29
  • 1
    It is possible to use App Engine and Datastore independently in multiple locations and to replicate Datastore changes via PubSub. I experimented with this concept here with this: https://github.com/leighmcculloch/google-cloud-appengine-datastore-replicator – Leigh McCulloch Jan 07 '19 at 06:16
  • @DanMcGrath Any update on doing cross region Datastore duplication (with consistency considerations) in 2023? – Micro Jul 20 '23 at 19:46