23

We've written a framework to test the performance of our Java application (none of the existing frameworks, eg JMeter, were appropriate). The framework produces various metrics, e.g. mean/min/max transactions per second.

We'd like each Jenkins build to display these metrics so that we can keep track of whether a commit has improved performance or not.

I can't figure out how to do this.

One idea is to modify our performance test framework to output a HTML file, then somehow make Jenkins display/link to it on the build results page.

Any advice gratefully received.

Phil Harvey
  • 1,160
  • 4
  • 13
  • 19
  • Best quick hack is to dump html in your job description and use relative links to the last build etc. E.g. with performance plugin `` – KCD Apr 05 '13 at 03:37

3 Answers3

13

The Peformance Plugin can show the results of JMeter and JUnit test in the nice, graphical fashion. And on the plugin page there is a description on how to use it.

This is an open-source plugin hosted on GitHub. The JUnit and JMeter parser are already there, but You can implement your own just by subclassing PerformanceReportParser. It's pretty easy and you can just fork the repo and start your implementation.

Christopher Orr
  • 110,418
  • 27
  • 198
  • 193
Łukasz Rżanek
  • 5,786
  • 27
  • 37
  • If we were using JUnit or JMeter to run our tests then that plugin would be useful. But our performance tests are entirely home-grown so I don't think that plugin will help us. – Phil Harvey Feb 15 '12 at 09:53
  • 2
    Yes, but it's open source. Just modify the plugin to your own needs! – Łukasz Rżanek Feb 15 '12 at 10:12
  • any sample implementations ? like to have additional inputs other than just Jmeter parameters ? – Njax3SmmM2x2a0Zf7Hpd Nov 05 '12 at 17:40
  • 1
    If using other tools the [JTL format](http://wiki.apache.org/jmeter/JtlTestLog) is simple to emulate, for each `httpSample` `t` is response time, `ts` is milliseconds since epoch, `s` is boolean for success, and `lb` is your URI or similar grouping label – KCD Apr 05 '13 at 03:32
7

I agree that it is hard (if not impossible) to squeeze all the information into standard formats, like JUnit. They are good for quick identification of problems. Once you know there is a problem - you need more information that is usually free-form or custom-formatted to fit your particular needs. So we use both: JUnit that can be immediately processed by Jenkins to decide if the build is stable or not, draw the nice trend graph, etc. We also produce an HTML report that is much more detailed.

Now to your immediate question: you can simply archive your HTML file as an artifact (there is a standard post-build step to do that). Then a link to it will be displayed among the artifacts for the build. There are permalinks to the latest artifacts and latest successful build artifacts:

http://[server]/job/[job_name]/lastCompletedBuild/artifact/foo.html
http://[server]/job/[job_name]/lastSuccessfulBuild/artifact/foo.html

You may bookmark those links and have quick and easy one-click access to your results.

malenkiy_scot
  • 16,415
  • 6
  • 64
  • 87
  • 1
    This will not give you real comparison between different builds but rather the immediate report. In order to really comprehend the changes of performance test between builds one have to build up the data somewhere, e.g. in DB. – Łukasz Rżanek Feb 15 '12 at 16:07
  • @ŁukaszRżanek: sure, but that's how I understood the question: HTML is custom-made. – malenkiy_scot Feb 15 '12 at 16:13
  • Therefore it would be easy to parse the result - an author of those HTML files knows them pretty well I guess. – Łukasz Rżanek Feb 15 '12 at 16:22
  • But the plugins not necessarily can display the metrics in a form that puts all the results together as wanted. – malenkiy_scot Feb 15 '12 at 16:26
  • Did you actually took a glance at the plugin page? – Łukasz Rżanek Feb 15 '12 at 18:38
  • @ŁukaszRżanek: and I use some of them myself. – malenkiy_scot Feb 15 '12 at 19:24
  • So I use the plugin myself. With JMeter and JUnit. And it's really got all the information you might actually want with timeline between builds and statistics. The guys that made that one really knew what they were doing. So I can risk saying it's way, way better then simple HTML pages if you want to compare how the performance was changing over time... – Łukasz Rżanek Feb 15 '12 at 19:39
  • There is an alternative way of looking at it: you've got used to the way they provide the information, and that's fine. If someone else has gotten used to some other way, it will be extremely inconvenient to them to abandon that way and switch to something else. Especially if we are talking about a whole team here, and not just one or two people. – malenkiy_scot Feb 15 '12 at 19:49
  • But then you're killing the whole purpose of using performance test - compare the results over time, setting some minimum and alert when they're breaks, aggregate for future reference. And again - if you're not evolving and making things better, you're taking step back at the time. But that's me and my need to change things and make them better... – Łukasz Rżanek Feb 15 '12 at 21:20
  • @ŁukaszRżanek, this is not a discussion forum. I think everybody has gotten your point by now. – malenkiy_scot Feb 16 '12 at 07:16
  • True. Cheers! But it was fun :) – Łukasz Rżanek Feb 16 '12 at 08:13
6

You could use the HTML Publisher Plugin to publish the resulting HTML page. That would be pretty straightforward.

If you want better integration you could try to create output that follows the same format JMeter produces, and use the Performance Plugin.

For best result you could take Łukasz advice and modify the Performance Plugin to your needs. That requires the most effort on your part, of course.

pushy
  • 9,535
  • 5
  • 26
  • 45