26

I wanted to test python27 on appengine so I have migrated my app from python25. Performance got more than 2x slower for every request! Then I've returned to python25 and performance is again as it was before. Here is a picture:

enter image description here (milliseconds/request) (cgi handler python 27, then python25)

My app uses Werkzeug, Jinja2, and memcache is used quite alot. What reasons can cause such a dramatic decrease in performance? Or is it just because python2.7 on appengine is still in beta?

Some details about application:

It is quite simple online shop. There are some deferred tasks with pdf generation however these don't affect overall graph much because the front page gets most hits. Nearly everything is memcached. It takes up to ~0.8 sec with empty cache to load a page with python 2.5. Non-cached pages takes long to load mainly because there are many db queries. Cached pages load in 60~100 ms. Average load time is ~150 ms. With python 2.7 performance is terrible. Non-cached pages takes 2+ secs to load. Cached pages load in 200+ ms.

Unfortunately I don't have any profiling data and I can't tell what exactly slows things down in python 2.7.

My numbers for page-load time are collected from live page which serves ~10 req/sec and 1 resident python25 instance easily deals with this load.

I have also tested python 2.7 with wsgi and threadsafe:yes, but performance improved just a little compared to python 2.7 and cgi.

Ski
  • 14,197
  • 3
  • 54
  • 64
  • I don't have any idea why it would be that slow, but they might be testing out a lot of stuff since python27 is still experimental in app engine. – bigblind Dec 03 '11 at 22:05
  • 1
    You haven't told us anything at all about your app or what it does, or what steps you've taken to diagnose. How could we possibly know what the problem might be? – Nick Johnson Dec 06 '11 at 23:28
  • i'm seeing the same thing. One bottleneck I identified was memcache api calls were slower over all for the requests I was testing, but I wouldn't be surprised if it was every api call. – Tom Willis Dec 21 '11 at 03:25

6 Answers6

17

Somewhere on Usenet I read a statement like this from Google "he Python 2.7 runtime is slower than the Python 2.5 runtime in some cases and faster in others. We aren't publicizing the reasons why at this point.". Seems nobody has found so far a scenario where 2.7 is faster than 2.5 thus ...

I read into this

  1. Google is aware of a preformace issue.
  2. They not sure how to handle it.

My profiling indicates that python 2.7 multithreaded applications spend ca. 35 % of their time in {method 'acquire' of 'thread.lock' objects} - and seemingly that happens in googles RPC code. There are also indication that importing has serious locking issues.

Basically you can't do anything about it except waiting for the AppEngine to fix it. See also this comprehensive documentation about the slowdown.

Factors which almost certainly don't play a significant role in this are:

  • uploading pyc files (the GAE infrastructure does that for you)
  • the server provisioning (GAE is such a massive scale and even in closed beta 2.7. was slow)
  • WSGI vs. CGI (wouldn't explain such a huge performance hit)

The ugly thing is that Google did a massive price hike in two steps but told us "by using multithreaded python 2.7 you can run so much more effective that the new prices don't look so bad". Unfortunately the Python 2.7. runtime is still labeled "experimental" and doesn't deliver production quality performance.

max
  • 29,122
  • 12
  • 52
  • 79
  • I now read several "We aren't publicizing the reasons for the sped issues" comments by AppEngine staff. So there really seems to be some fundamental issue. – max Jan 06 '12 at 18:36
  • 1
    Thanks mdorseif - I've found that {method 'acquire' of 'thread.lock' objects} takes up to 50% of the time in some of my calls in GAE...6 months on and this still seems to be an issue (even though 2.7 is not experimental anymore). Has anyone heard any updates from Google on this? Also, you mention that there is indication that importing has serious locking issues, is there anything that can be done about that by developers experiencing this issue? – Stin Apr 22 '12 at 17:15
  • The documentation linked in this answer indicates that Google deployed a fix on Jan 28 '12 and the problem no longer exists. – Julian Go Oct 26 '12 at 14:52
5

Python 2.7 support is still experimental. One aspect of being new and experimental is that it hasn't had the kind of performance baking and tuning that Python 2.5 has.

Dave W. Smith
  • 24,318
  • 4
  • 40
  • 46
  • Thank you for Your answer. Probably it was naive of me to expect something better from yet experimental feature. I hope to see python2.7 getting close to 2.5 in near future! – Ski Dec 05 '11 at 08:02
  • 2
    But on the other hand Google suggest to use python27 to get around there massive last price hike on 2011-12-01 ... – max Dec 06 '11 at 19:48
  • 1
    Python27 supports multiple concurrent requests, allowing you to handle the same load with fewer instances, and thus a smaller bill for instance hours. – Dave W. Smith Dec 06 '11 at 22:24
  • 2
    Dave, when each request takes 10 times as long, 8 concurrent requests per instance don't reduce my bill. Also as you say python27 is "experimental" so it shouldn't be used in production at all (if you believe conventional wisdom). So you got massive price hike or you have to use an experimental feature which is of shaky quality - and might not even reduce your bill :-( – max Dec 08 '11 at 06:57
  • 1
    If your requests are taking 10 times as long, we'd be really interested in what you're doing. That's pretty far out of line with what I'm seeing. – Dave W. Smith Dec 08 '11 at 17:52
3

Did you migrated your application to use WSGI instead of CGI when run on python 2.7?

It is possible that CGI interface is just a wrapper around WSGI that is enabled for 2.7 now.

lig
  • 3,567
  • 1
  • 24
  • 36
  • 1
    No I didn't migrate to wsgi. But your post motivated me to test python27 with wsgi and `threadsafe:yes` so I am going to test that now and I hope to have performance differences tomorrow. – Ski Dec 06 '11 at 16:22
  • There was tomorrow yesterday. What about your performance testing? – lig Dec 09 '11 at 13:50
  • Well there isn't much to say, because I couldn't see a difference. If there was an improvement it could not be higher than ~15% (python27 wsgi with `freadsafe:yes` compared to cgi). I was writing my application for cgi, and kept bootstrapping as light as I could, maybe that is why I don't see any improvement with multithreading. Resume is: python27 is terribly slow at the moment but I hope it will become faster when it is out of experimental stage. – Ski Dec 09 '11 at 15:48
  • 1
    @Skirmantas thanks for the info. P.S.: Please, speak that python2.7 on GAE is terribly slow, not just python2.7, because it is much faster usualy:) – lig Dec 09 '11 at 20:40
1

By swtiching to python2.7 my app performance under load is 3x worse.

With 2.5:

Connection Times (ms) min mean[+/-sd] median max Connect: 36 48 15.4 41 109 Processing: 685 3010 1893.3 2657 9255 Waiting: 685 3009 1893.3 2656 9255 Total: 725 3058 1900.5 2711 9333

Percentage of the requests served within a certain time (ms) 50% 2711 66% 3287 75% 3896 80% 4521 90% 6146 95% 7078 98% 7934 99% 8413 100% 9333 (longest request)

With 2.7:

Connection Times (ms) min mean[+/-sd] median max Connect: 35 46 11.4 41 96 Processing: 1076 7614 4190.5 6711 32284 Waiting: 1075 7614 4190.5 6711 32283 Total: 1124 7660 4195.5 6764 32353

Percentage of the requests served within a certain time (ms) 50% 6764 66% 7790 75% 8751 80% 9392 90% 10844 95% 13139 98% 25219 99% 27259 100% 32353 (longest request)

Reinaldo
  • 11
  • 1
0

A quote from google itself

Experimental!

The Python 2.7 runtime is an experimental, innovative, and rapidly changing new feature for App Engine. Unfortunately, being on the bleeding edge means that we may make backwards-incompatible changes. We will inform the community once the Python 2.7 runtime is no longer experimental.

They haven't really made it for production use yet , Its like beta testing .

So keep your application in python2.7 till the experimental phase is over.

You can also try uploading only the compiled .pyc files because python27 runtime supports bytecode upload

Gautam
  • 7,868
  • 12
  • 64
  • 105
-1

As far as I know, Python 2.7 should be faster than 2.5. However, there are some factors that can influence speeds:

  • The way the binary is compiled;
  • Whether your libraries (such as Memcache) are compiled as C(++) or Python. A C++ module is, of course, faster than the Python equivalent;
  • The server it's on - I haven't ever used App Engine but I'd assume that a server only runs either Python 2.5 or Python 2.7, since mixing them would be a waste of resources. If the 2.7 servers are used a lot more than the 2.5 servers and App Engine hasn't compensated for this, you'll notice the performance go down as well.

These are the first 3 things that I came up with, but there are a lot of factors. Even the weather could in theory affect performance.

Tom van der Woerdt
  • 29,532
  • 7
  • 72
  • 105