So I have a Flask socket IO application start with Gunicorn with the worker class : geventwebsocket.gunicorn.workers.GeventWebSocketWorker and with 1 worker. And, I have find that in a particular case i am getting a [critical] WORKER_TIME_OUT and the API die ( there is probably other case ). I was able to reproduce the issue by doing that :
def test_get_all(self):
pool = ThreadPool(3)
entity_route = [self.API_ROUTE, self.API_ROUTE, self.API_ROUTE]
pool.map(self.get_entity, entity_route)
def get_entity(self, route):
rest_client.post(route, json={
"email": 'DEFAULT_USER_EMAIL',
"password": DEFAULT_USER_PASSWORD
}, status_code=200).json()
So i am calling the API_ROUTE 3 times in parallel. Inside the controller of API_ROUTE I am making a function call that update a field in an entity in the PSQL database and in a document in the Elastic Search instance. At that moment, The API froze, and after the default 30 sec timeout die. If i comment the call to ES or the PSQL. It pass no problem. I try to use the worker class eventlet and it fix it, but then other route failed, because it seem responses get mixed up. So i am not sure what worker class to use, because i need the web socket functionality.
I also try to use a lock around the function that call Elastic Search and PSQL. But it still fail. Something like this :
from gevent.threading import Lock
lock = Lock()
self.lock.acquire()
entity.update(**data)
self.lock.release()
If someone could point to me how to setup a monitoring on the greenlet with gunicorn and also explain to me what is happening it would be great. Also, the application is running in docker and kubernetes ( minikube locally )
Thank you !!