App Engine Flexible can't use datastore with websockets example

Question

I tried adding the websockets example project to the datastore project and the websockets work but when a page queries the datastore or tries to put a new entity I get a 502 response. In the logs it shows a critical error on the service worker. If I remove the websocket code the datastore code works as intended. The only difference I can see is the entrypoints for the app samples slightly differ

the websocket sample uses

entrypoint: gunicorn -b :$PORT -k flask_sockets.worker main:app

while the datastore sample uses

entrypoint: gunicorn -b :$PORT main:app

websocket sample https://github.com/GoogleCloudPlatform/python-docs-samples/tree/master/appengine/flexible/websockets

datastore sample https://github.com/GoogleCloudPlatform/python-docs-samples/tree/master/appengine/flexible/datastore

score 2 · Answer 1 · answered May 10 '19 at 21:28

The problem appears to be that GRPC (the default transport mechanism of the Cloud Datastore client) is not compatible with gevent. Aside from using a different websockets framework, you can work around the issue by activating grpc's gevent compatibility patch, using the following code:

import grpc.experimental.gevent as grpc_gevent
grpc_gevent.init_gevent()

score 2 · Answer 2 · answered May 13 '19 at 23:09

As a complement of Andrew's answer, you can extend the gunicorn worker class to run gRPC applications.

# gevent_grpc_worker.py
from gunicorn.workers.ggevent import GeventWorker
from grpc.experimental import gevent


class GeventGrpcWorker(GeventWorker):
    def patch(self):
        super(GeventGrpcWorker, self).patch()
        gevent.init_gevent()
        self.log.info('patched grpc')

# config.py for gunicorn
import multiprocessing
from gevent_grpc_worker import GeventGrpcWorker


# http://docs.gunicorn.org/en/stable/design.html#how-many-workers
workers = multiprocessing.cpu_count() * 2 + 1
worker_connections = 10000
# Use an asynchronous worker as most of the work is waiting for websites to load
worker_class = '.'.join([GeventGrpcWorker.__module__,
                         GeventGrpcWorker.__name__])
timeout = 30

Then start your managed application by:

gunicorn -c config.py app:app

score 0 · Accepted Answer · answered Feb 05 '19 at 11:02

As you said it seems there is a problem with the flask_socket.worker, I have test it and it does not work with the datastore client.

I have tried with the Flask-SocketIO framework using the eventlet worker and the datastore queries work fine.

entrypoint: gunicorn -b :$PORT --worker-class eventlet -w 1 main:app

Also you need to add the eventlet module in the requirements.txt file eventlet==0.24.1

The downside of this is that it breaks the compatibility with the websocket code so you need to rewrite this part. Keep in mind that code samples are just intended to show in a few lines how to use the Google Cloud products and copy-paste them without modifying the configuration undelied in the app.yaml is not a good idea.

App Engine Flexible can't use datastore with websockets example

3 Answers3