20

Currently our Jenkins server only displays a history/graph for the overall number of passed/skipped/failed tests - I'm assuming that's the behavior out of the box.

If you select a single test, you'll get information for how long the test was failing (assuming it did fail).

However, we'd like to see is a history for that single test across the different builds to identify whether the test has been failing in the past (and when) even though it just passed. If you find a build where it failed, you could click on it, and investigate what might have caused the failure; if it passes again, you could check whether something actually fixed the test, or whether it was failing randomly all along.

Is this something that can be done somehow through the config, or do we need an additional plugin for this? If yes, which one?

Not sure if this makes much difference, but we're using Java (Maven) & TestNG (Surefire).

mac
  • 2,672
  • 4
  • 31
  • 43
  • Are you using JUnit or TestNG? What about the plugins for those two? This doesn't solve your problem? – mac Oct 29 '15 at 05:05

3 Answers3

8

Both the TestNG plugin and the JUnit plugin will actually display history of the test results.

You just need to pick a given result and then:

  • For JUnit click on "History" on the left side, and
  • For TestNG click you will see the history in the graph above the result. You can just click on the bars in the bars to see the older results, and also if you click closer to the edge, the scope of the test results will adjust
mac
  • 2,672
  • 4
  • 31
  • 43
  • Sadly the JUnit History view does not seem to be what we're looking for. – Ed Brannin Jul 30 '15 at 16:06
  • What are you looking for that the JUnit (History) does not provide? – mac Jul 31 '15 at 17:25
  • The JUnit history will show me if Test X has been failing for the past 10 builds. That's great. But if Test Y fails on every other build, then I'd like a way to see that it has a rough history even if its maximum failure-streak is 1. – Ed Brannin Aug 03 '15 at 13:56
  • The plugin should show you the history of the last x executions, as both graph and table, see example here: http://tiernok.com/LTDBlog/ContinuousDelivery/unittest_history_lg.png I'm not sure I understand what you'd like to see that this doesn't show. – mac Aug 03 '15 at 22:17
  • 1
    For _each specific test_, how often has it failed in the past? The picture you linked is a history of the entire test suite's runtime/pass/failure counts, but does not provide a deep history of the stability of any specific test cases. This is more useful for legacy test suites that are not quite at 0-failures-per-build, so there's always _some_ failure and we want a better idea of what's recurring and what's totally new. – Ed Brannin Aug 04 '15 at 17:27
  • @mac - How can we store test history without Jenkins ? Is Jenkins the best way to store it ? – MasterJoe May 30 '18 at 17:32
5

The Test Results Analyzer plugin does the job for me. There appears to be other suitable plugins out there as well.

https://wiki.jenkins-ci.org/display/JENKINS/Test+Results+Analyzer+Plugin

WoodenKitty
  • 6,521
  • 8
  • 53
  • 73
1

Does the Static Code Analysis plugin help?

Slav
  • 27,057
  • 11
  • 80
  • 104