0

In my GAE app, there are some tasks which were being performed in taskqueues, asynchronously. These tasks involve fetching data from an API and processing it. As the app matured, the data grew in size so the taskqueue limit of 10 minutes started to break in some cases. To fix this I moved the app to GAE modules architecture and moved the asynchronous execution to a backgrouindThread. This works but the execution has become a lot slower, typically taking 8x time. What is the reason behind this? What is the solution/workaround for this, if any?

EDIT:

I suppose it's because of poor multithreaded performance of python. Is it true? Can anyone confirm this?

0xc0de
  • 8,028
  • 5
  • 49
  • 75
  • What do you see in the logs? Is it possible that startup time explains the difference? – Andrei Volgin Oct 09 '14 at 06:36
  • @AndreiVolgin As far as I have seen, it's the actual execution what takes time and not any delay in starting the thread. – 0xc0de Oct 09 '14 at 06:39
  • @AndreiVolgin, Yes, the highest configuration to be specific. One difference though, the later added module has basic scaling. Just curious, does the number of instances change anything regarding a single execution here? I think it won't but just being sure.... – 0xc0de Oct 09 '14 at 06:46
  • Only a startup time, but you say it's not the cause. – Andrei Volgin Oct 09 '14 at 06:56
  • If you set a low max_instances it can bottle neck execution. If you send 10 requests, but max at 5 instances it will take 2ce as long to run all the tasks. – Ryan Oct 09 '14 at 21:51
  • @Bruyere: Yep I understand that, thanks. But this performance degrade is observed per request :-/ . – 0xc0de Oct 10 '14 at 06:48

0 Answers0