ANT Jenkins unit test - count failed tests

Question

I would like to count the failed tests, and make a failure only once when all tests completed. These tests are run by Jenkins every night, and reporting the results if there are failures or errors. The problem is that I can't even start counting, because this has only failureProperty and errorProperty which can be true or false, but when it reaches the first failed or errorous test, it stops and fails. I didn't find good solution with google, they recommended me these properties, but they don't do what I need.

here is the code:

        <junit printsummary="true" fork="yes" forkmode="once" 
showoutput="false" haltonfailure="no" maxmemory="1024m" 
errorProperty="test.failed" failureProperty="test.failed">
            <classpath refid="junit.classpath" />
            <batchtest fork="yes" todir="${junit.dir}/raw" >
                <formatter type="xml" />
                <fileset dir="${classes.dir}">
                    <include name="**/*Test.class" />
                    <exclude name="*ear*/**"/>
                    <exclude name="**/Base*.class" />
                    <exclude name="**/JNDI*.class" />
                </fileset>
            </batchtest>
        </junit>
    <fail message="At least one test failed or has errors, please check test results in Jenkins to see details!" if="test.failed" />

Do I miss something important? It seems haltonfailure="no" parameter is not working in this case.

Thanks in advance if you can help me!

possible duplicate of [How can I have the Ant JUnit task run all tests and then stop the rest of the build if any test has failed](http://stackoverflow.com/questions/4038547/how-can-i-have-the-ant-junit-task-run-all-tests-and-then-stop-the-rest-of-the-bu) — Miquel, Feb 08 '13 at 14:24
I found out what is the problem. the _errorProperty="test.failed" failureProperty="test.failed"_ works with _fail_ I have more bunches of unit tests, so the script survives the first one only! I was a bit blind, but this gives me an idea. Thank you all! — CsBalazsHungary, Feb 08 '13 at 14:33
Just for record, I made a global variable which is false, and I switched to true if a test fails. Then where I ran all the separate tests I called the fail ant method. — CsBalazsHungary, Feb 08 '13 at 14:47
Oh hey, I just found a duplicate answer to yours, which seems to work. Check the [accepted answer here](https://stackoverflow.com/questions/4038547/how-can-i-have-the-ant-junit-task-run-all-tests-and-then-stop-the-rest-of-the-bu) — Miquel, Feb 08 '13 at 14:24

score 1 · Answer 1 · answered Feb 08 '13 at 14:05

1

You need to generate a report using ant's junitreport. This outputs an xml file that jenkins will nicely count and display for you, together with evolution graphs and so on if you are interested.

answered Feb 08 '13 at 14:05

Miquel

15,405
8
54
87

Thanks it would be good, but in this case can I invoke a in ant? I mean Jenkins needs this to send the warning mail. The point why I need this because I want to send a mail when it fails. – CsBalazsHungary Feb 08 '13 at 14:08
@CsBalazsHungary Jenkins, by default will not fail the build on test failure. Rather, it will consider it "unstable". So, you get three possible outcomes (all of which can send an email if you want): blue=success, yellow=unstable (tests failed) and red=failed – Miquel Feb 08 '13 at 14:10
hmm but I edit the question, it is not clear what I really want with this. :) – CsBalazsHungary Feb 08 '13 at 14:10
Ok :) Also, check out what I mean for example [here](https://builds.apache.org/job/ACE-trunk/) where some balls are yellow (failed tests) and there's a test history graph – Miquel Feb 08 '13 at 14:11
You are right, until I started to play with the errorproperty and failureproperty, it made all the tests and ant build was success, so it almost does what I want, just I need to artificially kill the ant task when I see at least one unit test which fails or errorous. – CsBalazsHungary Feb 08 '13 at 14:13
yes, just suggested by David W. I didn't adjust first haltonerror, but even if I gave it a "no" value it didn't help. – CsBalazsHungary Feb 08 '13 at 14:19
Aaah I see. You want to continue running the tests, but remember that one failed. Ok. In maven, not that it helps you, instead of using the normal junit runner (surefire) you use failsafe, which does exactly that. – Miquel Feb 08 '13 at 14:22

score 1 · Answer 2 · answered Feb 08 '13 at 14:12

Set the JUnit properties haltonerror and haltonfailure to false. Then use various reporting tools to parse the generated report. The report can be in XML format, and you can use JUnitreport to parse the file. I prefer Jenkins which will show me each build, and the number of failed JUnit tests, and all sorts of nice reports. Others use various XSLT programs (which is really what JUnitreport is).

Word 'o Caution: If you use JUnitReport, you might need a few extra jars in your Ant's lib directory. I haven't used JUnitReport in a while, but I remember I needed Xalan, but that is something that now comes with Java.

sadly if I use errorproperty and failureproperty and then fail using these properties, the unit tests stop running on first occurance of failure or error. But thanks, it will be useful if there is no way to kill ant task on the end — CsBalazsHungary, Feb 08 '13 at 14:17

score 1 · Answer 3 · edited May 23 '17 at 12:02

1

Also check this answer from another similar question on how to configure your junit target both for execution and report generation. Once you generate the report, configure with the JUnit test report generation post build action.

Also have you tried using printsummary="withOutAndErr" and checked? This works completely fine for me.

edited May 23 '17 at 12:02

Community

1
1

answered Feb 08 '13 at 14:21

Shiva Kumar

3,111
1
26
34

ANT Jenkins unit test - count failed tests

3 Answers3