20

Our Python application (a cool web service) has a full suite of tests (unit tests, integration tests etc.) that all developers must run before committing code.
I want to add some performance tests to the suite to make sure no one adds code that makes us run too slow (for some rather arbitrary definition of slow).
Obviously, I can collect some functionality into a test, time it and compare to some predefined threshold.

The tricky requirements:

  1. I want every developer to be able to test the code on his machine (varies with CPU power, OS(! Linux and some Windows) and external configurations - the Python version, libraries and modules are the same). A test server, while generally a good idea, does not solve this.
  2. I want the test to be DETERMINISTIC - regardless of what is happening on the machine running the tests, I want multiple runs of the test to return the same results.

My preliminary thoughts:

  • Use timeit and do a benchmark of the system every time I run the tests. Compare the performance test results to the benchmark.
  • Use cProfile to instrument the interpreter to ignore "outside noise". I'm not sure I know how to read the pstats structure yet, but I'm sure it is doable.

Other thoughts?

Thanks!

Tal.

Tal Weiss
  • 8,889
  • 8
  • 54
  • 62

2 Answers2

5

Check out funkload - it's a way of running your unit tests as either functional or load tests to gauge how well your site is performing.

Another interesting project which can be used in conjunction with funkload is codespeed. This is an internal dashboard that measures the "speed" of your codebase for every commit you make to your code, presenting graphs with trends over time. This assumes you have a number of automatic benchmarks you can run - but it could be a useful way to have an authoritative account of performance over time. The best use of codespeed I've seen so far is the speed.pypy.org site.

As to your requirement for determinism - perhaps the best approach to that is to use statistics to your advantage? Automatically run the test N times, produce the min, max, average and standard deviation of all your runs? Check out this article on benchmarking for some pointers on this.

rlotun
  • 7,897
  • 4
  • 28
  • 23
2

I want the test to be DETERMINISTIC - regardless of what is happening on the machine running the tests, I want multiple runs of the test to return the same results.

Fail. More or less by definition this is utterly impossible in a multi-processing system with multiple users.

Either rethink this requirement or find a new environment in which to run tests that doesn't involve any of the modern multi-processing operating systems.

Further, your running web application is not deterministic, so imposing some kind of "deterministic" performance testing doesn't help much.

When we did time-critical processing (in radar, where "real time" actually meant real time) we did not attempt deterministic testing. We did code inspections and ran simple performance tests that involved simple averages and maximums.

Use cProfile to instrument the interpreter to ignore "outside noise". I'm not sure I know how to read the pstats structure yet, but I'm sure it is doable.

The Stats object created by the profiler is what you're looking for.

http://docs.python.org/library/profile.html#the-stats-class

Focus on 'pcalls', primitive call count, in the profile statistics and you'll have something that's approximately deterministic.

S.Lott
  • 384,516
  • 81
  • 508
  • 779
  • For my system, `pcalls` is pretty deterministic. I have a test in place to detect if `pcalls` changes, and it reports by what percent. A test failure can then be a jumping off point to investigate what has changed and whether the performance impact is acceptable. – Peter Wood Apr 18 '16 at 14:27