14

I'm writing a web-application in Python, I haven't decided if I want to use Flask, web.py or something else yet, and I want to be able to do profile on the live application.

There seems to be very little information on how you go about implementing the instrumentation to do performance measurement, short of doing a lot of print datetime.now() everywhere.

What is the best way of going about instrumenting your Python application to allow good measurements to be made. I guess I'm looking for something similar to the Stackoverflow teams mvc-mini-profiler.

Simon
  • 520
  • 1
  • 4
  • 13
  • 3
    http://docs.python.org/library/profile.html – Wooble Jun 21 '11 at 17:54
  • I don't really like the fact the profiling of individual functions done the way the standard profile suggests. It seems mostly useful if you expect the application to terminate at some point. I do see how it could be done elegantly with decorators, so perhaps it's worth looking into writing some test for. – Simon Jun 21 '11 at 18:08

4 Answers4

8

You could simply run cProfile tool that comes with Python:

python -m cProfile script.py

Of course, you would have to create the script.py file that would execute the parts of the code that you want to test. If you had some unit tests, you could also use that.

Or you couse use:

import cProfile 
cProfile.run('foo()')

to profile it from foo entry point.

rafalotufo
  • 3,862
  • 4
  • 25
  • 28
  • I would really prefer having the profiler code in the function, rather than "wrapping" the function in the profiler. Also doesn't it make it harder to turn profiling on and of for parts of the applications? – Simon Jun 21 '11 at 18:37
  • Turns out that despite my reluctance to use the standard profiler, this is the correct answer, but with a small modification. Someone wrote this: http://mg.pov.lt/profilehooks/ which allows you to add a decorator to the functions you want to profile. It's not perfect, as it fails to collect everything like Stackoverflow mvc profiler does, but I guess that's the price of having something a bit more generic. – Simon Jun 21 '11 at 21:23
  • Thanks for the comment, I didn't know about profilehooks. Cheers – rafalotufo Jun 21 '11 at 23:21
4

Amir Salihefendic wrote a short (150 LOC) RequestProfiler, which is described in this blog post:

I haven't tried it, but since it is a WSGI middleware, it should be somewhat pluggable.

miku
  • 181,842
  • 47
  • 306
  • 310
  • Looks intersting, a bit limited perhaps and I don't really understand how it manages to track database and memcache queries, but it looks like a quick way for locating bottlenecks. – Simon Jun 21 '11 at 18:17
  • I haven't looked at the source myself - I trust the author ;) However, I hope other answers appear to this question, since I have an interest in this topic myself (maybe the question title can be a bit more focused e.g. on WSGI profiling). – miku Jun 21 '11 at 18:22
  • It wasn't meant to be WSGI profiling. I actually don't care for solution that is WSGI only, I wan't to be able to use the same tools for web and ordinary applications. – Simon Jun 21 '11 at 18:32
0

Use New Relic's Free monitoring system. You simply install an agent on the server and point to your flask init.py file. Once you run the application with proper agent setup, you will start seeing application metrics in see New Relic's online dashboard called APM. By default it will show you graphs of your application's throughput (QPS/RPM), app response time, top transactions, error rate, error stack trace if any(eg for 500 error), calls to external services etc. In addition you can monitor your System stats too.

user2699006
  • 17
  • 1
0

You can just use a general purpose web application performance tool, such as httpperf. This works using an external client and works with any framework since it works against a standard interface (HTTP). Therefore it tests the full stack performance.

Keith
  • 42,110
  • 11
  • 57
  • 76
  • But it won't tell me where time is spent, just that a given http query is slow. – Simon Jun 21 '11 at 19:40
  • But will help you compare framework performace that you seemed to imply that you want to do. Otherwise, you can use the profiling tools available that you say you don't actually want to use for some reason. – Keith Jun 21 '11 at 20:44
  • May that was my mistake to talk about web frame works, it was meant as "I want something that supports any framework". I don't want to use the standard Python profiler, because it doesn't seem like something you would like to run constantly in a production system, but more something you use to measure performance before deploying. – Simon Jun 21 '11 at 21:05