How do I force dev_appserver to reference local data?

Question

I've been using dev_appserver for years for local Python 2.7 development. The datastore is created locally and just works as expected. I'm now migrating to Cloud Datastore and the default behavior appears to be that it accesses my production datastore. Not at all what I expected, and I cannot get dev_appserver to look locally rather than to my cloud data.

The docs: https://cloud.google.com/datastore/docs/tools/datastore-emulator

... tell me to set my env vars as follows:

DATASTORE_DATASET=<my dataset>
DATASTORE_PROJECT_ID=<my project id>
DATASTORE_EMULATOR_HOST_PATH=localhost:8081/datastore
DATASTORE_HOST=http://localhost:8081
DATASTORE_EMULATOR_HOST=localhost:8081

I'm invoking dev_appserver as follows:

    python3 /home/ml/google-cloud-sdk/bin/dev_appserver.py \
    ~/appengine/apps/$1 \
    --host=dev6 \
    --enable_host_checking=false \
    --port=8082 \
    --admin_host=dev6 \
    --admin_port=8002 \
    --application=<my app> \
    --support_datastore_emulator=true \
    --enable_sendmail \
    --smtp_host=dev6 \
    --smtp_port=10025 \
    --api_port=37063

... and I've tried this also without '--support_datastore_emulator=true'.

... yet my app is still accessing my production datastore.

Can anyone tell me how this is supposed to be configured in order to use a local datastore?

Did you first start the emulator by running ```gcloud beta emulators datastore start```? — NoCommandLine, Mar 28 '23 at 18:24
Yes. I've tried with and without the emulator running. Same result -- my queries pull from the production (cloud) datastore. I've run out of things to try :) — motoaddict, Mar 28 '23 at 22:26
If you're using the datastore client, it will pull from Production (based on Google's documentation, I'd say it's a bug). The only way I've been able to use local data is to use the bundled API — NoCommandLine, Mar 28 '23 at 23:13
Don't know what the "bundled API" is. I've posted how to make this work. — motoaddict, May 13 '23 at 16:10

score 0 · Answer 1 · edited May 19 '23 at 00:08

I found this very good article on Medium, which describes specifically how to get dev_appserver.py to play nice with the Cloud Datastore Emulator. The relevant part follows:

Running the Development Sever: dev_appserver.py

You’ll need two terminals for this. In the first do:

Initialize the Datastore emulator.

$ gcloud beta emulators datastore start

When you see that the start up has finished, you’ll see this message.

[datastore] 
[datastore] Dev App Server is now running.
[datastore]

Open another terminal and run the following command to see the emulator’s and active project’s environment variables.

$ gcloud beta emulators datastore env-init

The command above will give you an output as seen below. Save these for the next steps.

export DATASTORE_DATASET=<your-project-name>
export DATASTORE_EMULATOR_HOST=localhost:8081
export DATASTORE_EMULATOR_HOST_PATH=localhost:8081/datastore
export DATASTORE_HOST=http://localhost:8081
export DATASTORE_PROJECT_ID=<your-project-name>

Apply the variables to this terminal:

$ $(gcloud beta emulators datastore env-init)

This need to be done in any console you want to run dev_appserver.py. Without it the Datastore GUI will not connect to the Datastore emulator instance you've just opened; it will start another.

Pass the same variables to the Python virtual environment inside dev_appserver.py with the following call structure:

$ dev_appserver.py \
--application=<your-project-name> \
--env_var DATASTORE_DATASET=<your-project-name> \
--env_var DATASTORE_EMULATOR_HOST=localhost:8081 \
--env_var DATASTORE_EMULATOR_HOST_PATH=localhost:8081/datastore \
--env_var DATASTORE_HOST=http://localhost:8081 \
--env_var DATASTORE_EMULATOR_HOST_PATH=localhost:8081/datastore \
app.yaml

This is also very important. Without it the libraries inside the virtual environment will not point to your emulator; they will point to the real Datastore instead.

Celebrate! The Datastore GUI on localhost:8000 and your app libraries should both be pointing to the emulator you opened in step 9.

How do I force dev_appserver to reference local data?

1 Answers1