3

Unit tests are a great way to measure application functionality, but I'm wondering has anyone used them for some preliminary performance profiling?

What I'm talking about is running some profiling tool as part of an automated test run, saving performance results and then comparing them to some arbitrary value(s) so any bottlenecks would jump out. For example, I recently had a case that a Django tag library increased the template parsing time literally tenfold -- if the template parsing was built in unit tests it would be visible much sooner.

Are there any modules that would include this kind of measurement to a standard Django an/or Python setup? If not, do you have any particular suggestions or heads-up for writing my own? Thanks!

Berislav Lopac
  • 16,656
  • 6
  • 71
  • 80
  • Our CI monitors test time but profiling can cause considerable increasing of testing time. – oleg May 29 '13 at 11:31
  • Of course; it would be prudent to have a switch to enable/disable profiling when running tests. Care to share your solution? – Berislav Lopac May 29 '13 at 11:43

1 Answers1

2

I'll just describe what we're trying to do for estimating preliminary performance impact between releases. Not sure if it's the best approach but it works for us.

We have a bunch of tests in our Python/Django project. We are using nose as a test runner. As you may know, it has a built-in command-line option --with-xunit that dumps test results into an xml file in xUnit format (that is supported by jenkins CI by the way).

Each time we do a release, we're performing a test run and nose produces an xml file for us (and we store it in the code repository). Here's a part from it:

...
<testcase classname="prj.tests.functional.services.workflow.test_getCases.GetCases" name="test_full_access_ok"
          time="1.775"/>
<testcase classname="prj.tests.functional.services.workflow.test_getCases.GetCases"
          name="test_illegal_assigned_flag" time="0.008"/>
<testcase classname="prj.tests.functional.services.workflow.test_getCases.GetCases" name="test_illegal_comments"
          time="0.049"/>
...

As you may noticed, there is a time attribute for all test cases. So, what we do is we are just comparing test cases execution time by name against previous and current releases (basically, comparing two xml files).

Hope that helps.

alecxe
  • 462,703
  • 120
  • 1,088
  • 1,195