In my GAE app, there are some tasks which were being performed in taskqueue
s, asynchronously. These tasks involve fetching data from an API and processing it. As the app matured, the data grew in size so the taskqueue limit of 10 minutes started to break in some cases. To fix this I moved the app to GAE module
s architecture and moved the asynchronous execution to a backgrouindThread
. This works but the execution has become a lot slower, typically taking 8x time. What is the reason behind this? What is the solution/workaround for this, if any?
EDIT:
I suppose it's because of poor multithreaded performance of python. Is it true? Can anyone confirm this?