0

I am trying to migrate a Python 2.7 App Engine project from NDB to Cloud NDB as part of the migration process to Python 3.

After following the Cloud NDB migration instructions, just running the dev_appserver as before now results in accessing the cloud rather than the local datastore. I see Google's instructions for ensuring one accesses the local data, but I guess I don't understand how to use this in practice.

Assuming I have to use the datastore emulator to prevent this, I run dev_appserver with the flag --support_datastore_emulator true. This results in a successful conversion of my local datastore data into the sqllite format, but still queries the cloud.

I then set the required environment variables in app.yaml: DATASTORE_DATASET, DATASTORE_PROJECT_ID, DATASTORE_EMULATOR_HOST, DATASTORE_EMULATOR_HOST_PATH, DATASTORE_HOST (the values match the output of gcloud beta emulators datastore env-init). Running it complains that DATASTORE_APP_ID is not set, so I set it as well.

Everything now launches with a confirmation message that the emulator is being used, but trying to access the datastore results in "BadArgumentError: Could not import googledatastore. This library must be installed with version >= 6.0.0 to use the Cloud Datastore API." After installing that, I get a never-ending series of additional installation requirements and module conflicts... it's a mess, and this isn't listed in the documentation anyway.

How can I get dev_appserver (with or without the datastore emulator) to access local data rather than the cloud? Sadly, I have now spent days trying to make this work.

Dev93
  • 668
  • 1
  • 6
  • 18

2 Answers2

0

Try this (do it via terminal instead of specifying the environment variables in app.yaml).

  1. Open a terminal and follow the instructions for starting your datastore emulator

  2. After the emulator is up and running, in a separate terminal, run the dev_appserver.py command

If the above doesn't work, then 2 workarounds (have tested and confirmed these work)

  1. Repeat the above steps but start your python app without dev_appserver.py. I tested this with Flask and it worked (I tested this with our App but that's the underlying code for the feature that I used in the test)

  2. Second option is to use Bundled API for NDB (see docs for setting it up). If you go this route, then you don't need to explicitly start/enable the datastore emulator. This method will automatically use local datastore when you're not running in Production and will connect to production data when you're running on google cloud

NoCommandLine
  • 5,044
  • 2
  • 4
  • 15
  • really appreciate the response. I didn't list the many things I tried over the past few days, but I did try running the dev_appserver and the emulator in separate windows, to no avail (they connected, but queries still went to the cloud). I just tried your workaround 1, and trying to run the python app from Flask without invoking the dev_appserver just errors out, complaining about missing modules. And option 2 will only be viable once I complete the port to Python 3. – Dev93 Aug 05 '23 at 18:34
  • Are you able to share some of the missing modules it's complaining about? – NoCommandLine Aug 05 '23 at 18:41
  • With all my experimenting, @NoCommandLine, I have now completely bollixed this environment. Rather than try to recreate, I'm going to skip ahead to the full Python 3 implementation. But I'm already running into issues trying to understand best way to run/debug locally. Would love an answer to [this question on dev environment recommendations](https://stackoverflow.com/questions/76852788/how-to-test-run-debug-non-legacy-app-engine-project-using-pycharm-on-windows) – Dev93 Aug 07 '23 at 14:47
0

The first argument to

ndb.Client(gae_project_name)

is the project name. I set this to a different value when running locally to ensure that I don't accidentally access production.

new name
  • 15,861
  • 19
  • 68
  • 114