How to test/run/debug Non-Legacy App Engine project using PyCharm on Windows

Question

For years I have been developing for Python 2.7 App Engine using PyCharm on Windows. Works amazingly well, I can run or debug locally at the press of a single button within PyCharm. Love it!

As part of the forced migration to Python 3 App Engine, this beautiful combination no longer apparently works. Google docs say that its dev_appserver, which PyCharm runs behind the scenes, does not support development of Python 3 apps on Windows. NoCommandLine has developed a patch that may make this possible, but it’s not clear to me how robust this is, or how sustainable (assuming it works). Even if this would work, would PyCharm correctly call dev_appserver? PyCharm's docs say they always use the Python 2.7 runtime when creating a new project. It's not clear what this means, but right away the project complains about missing app.yaml keywords (which were previously required, and are now no longer permitted).

I would love to continue the single-button run/debug local testing using PyCharm. Is that possible?

I'm starting to think the best approach to development and local testing is to create a Flask project within PyCharm, and use PyCharm to run/debug using the Flask development server, separately running a local App Engine datastore emulator so it can access a local NDB. One "minor" issue with this is that I'd have to port numerous app.yaml static handlers to Flask to actually test a localhost web page.

Is there a better way? Please help me from having to rediscover the wheel...

Answer 1

Sharing this as a community wiki post.

@NoCommandLine stated that the patch is effective. For more details, refer to this post: Stack Overflow Response. You can also find relevant information in this response on the Google Cloud Community platform .

Accepted Solution

1. For Python 3 (this isn't happening for Python 2), dev_appserver.py works by creating a virtual env, installing the contents of your requirements.txt file and also installing gunicorn if your app.yaml doesn't include an entry point. It will then run your App with gunicorn. gunicorn is a Python WSGI HTTP Server for UNIX and so it won't work on Windows. I believe this is why dev_appserver.py for Python 3 isn't supported for Windows
2. But if you still want to use dev_appserver.py for Python 3 Apps on a Windows machine, you can check out a Patch we created (the patch essentially swaps out Gunicorn for Waitress when running your App on your dev machine).
3. Another option for your migration is to replace the Bundled APIs with their Cloud Library versions and then you can run your Apps the way you would normally run a Flask or Django App. What you'll then be missing would be testing your static handlers in dev environment and also testing multiple services (if you have multiple services).

It seems likely that this individual is also using PyCharm.

Answer 2

So you want to develop for Python 3 App Engine on Windows using PyCharm? Here's how it's done (order matters!)

1. Install a copy of the Google Cloud SDK for Python
2. Run gcloud components install app-engine-python and app-engine-python-extras
3. set CLOUDSDK_PYTHON=C:\Python\Python39\python.exe (or wherever your Python lives)
4. Using PyCharm create a Flask project, and set its run configuration to launch app/main.py (you will simply be running/debugging a Flask app, ignoring app.yaml completely for local dev)
5. In your main.py, you may need to duplicate any static handlers you have in your app.yaml
6. Copy the google folder from the App Engine SDK installation (e.g., C:\Users\XXX\AppData\Local\Google\Cloud SDK\google-cloud-sdk\platform\google_appengine\google) into your project's venv\Lib\site-packages (this will allow PyCharm to find ALL the google modules, since otherwise you will likely have 2 google module installations as described in the question)
7. Allow PyCharm to satisfy the requirements.txt appengine-python-standard>=1.0.0, installing more libraries into your project's venv, and overwriting some of the files you copied
8. In your project's venv\Lib\site-packages\google\appengine\runtime\thread_hooks.py, comment out the line importlib.reload(threading_module) so that PyCharm debugging will work

To use a local datastore, use the datastore emulator:

1. Include google-cloud-ndb in your requirements.txt
2. gcloud components install beta
3. You will also need a Java JRE (runtime engine) v11+ in your PATH
4. In your project's Run/Debug configuration, set environment variable DATASTORE_EMULATOR_HOST=localhost:8081 (or whatever port you are using)
5. Start the emulator via gcloud beta emulators datastore start before you run/debug your program from PyCharm

You can now develop and debug as normal... but you cannot locally access legacy bundled services.

If you want to do that, you will need to use dev_appserver, which is not officially supported on Windows. NoCommandLine has developed a patch that allows this, but while the patch will allow you to run, it will not allow you to debug within PyCharm. The dev_appserver also takes a LONG time to startup, even with the --python_virtualenv_path.