Failing Android JUnit tests, not breaking my Ant script like I expect?

Question

Failing JUnit tests, not breaking my Ant script like I expect?

My continuous integration server runs an Ant script, which calls something like: /tests/ant run-tests

My JUnit tests run, but with errors: run-tests: [echo] run-tests-helper. [echo] Running tests ... [exec] [exec] com.zedray.stuff.FooBarTest:.... [exec] com.zedray.stuff.FooBarTest:.....INSTRUMENTATION_RESULT: shortMsg=Some error in your code. [exec] INSTRUMENTATION_RESULT: longMsg=java.security.InvalidParameterException: Some error in your code [exec] INSTRUMENTATION_CODE: 0

The errors are OK, but my build script keeps going (eventually publishing my broken app to my testers - bad!). What I would expect is for the instrimentaiton to throw a build error, so my continuous integration server (TeamCity in this case) realises that something has gone wrong and reports a broken build. The "failonerror" is already set in the relevant macrodef, so I'm not sure what else I can do?

/tests/build.xml

Running tests ...

Any ideas/suggestions on how to fix this?

Regards Mark

Mark did you find what you needed here? Like does that work? — Greg Giacovelli, Oct 28 '10 at 21:57
[There's a filed bug about this](http://code.google.com/p/android/issues/detail?id=14241). It was reported in January. Unfortunately, the Android team doesn't read their bug DB. — Dan Fabulich, Apr 14 '11 at 02:10
When I had this problem I came up with this solution to solve it. https://github.com/curiousminds/teamCityAntTestRunner — BIT, Aug 20 '11 at 00:13

score 12 · Answer 1 · answered Sep 25 '12 at 14:53

I did it another way, because I am using the ant test target that is in the Android build.xml file. This target prints to the standard out, so I captured stndout into a file then queried the file, using this result to fail my task.

 <target name="run-acceptance-tests" depends="clean, debug, install" >

    <property name="log.file" value="acceptance_tests_standard_out.txt" />
    <!-- because we don't have control over the 'test' target (to check for passes an fails) this prints to standard out
         we capture standard out into a file and query this to see if we have any test failures, using this to pass/fail our task -->
    <record name="${log.file}" action="start" />
    <antcall target="test" />
    <record name="${log.file}" action="stop" />

    <!-- do other stuff -->

    <loadfile property="tests.output" srcFile="${log.file}" />

    <echo>Checking for failures</echo>
    <fail message="acceptance tests failed!" >
        <condition>
            <contains string="${tests.output}" substring="FAILURES" />
        </condition>
    </fail>

    <echo>acceptance tests passed!</echo>
</target>

Quite useful. :) And to help out others, if they aren't super-familiar with ant: You just paste this into your build.xml file and then run it using the command 'ant run-acceptance-tests'. — CasualT, Sep 27 '13 at 16:53
Simple and works, thanks! This was not working when i ran from a directory above the build.xml (record filename was relative to the current directory, but loadfile was looking relative to the build.xml). Replacing '' fixed this (as did using a full instead of relative path). — leorleor, Dec 20 '13 at 00:31

score 4 · Answer 2 · answered Apr 20 '11 at 14:54

I had the same problem, and I ened up customize the "run-tests" target in my own build.xml like this, and there is no need to change the original android sdk test_rules.xml

<target name="run-tests" depends="-install-tested-project, install"
            description="Runs tests from the package defined in test.package property">
    <echo>Running tests ...</echo>
    <exec executable="${adb}" failonerror="true" outputproperty="tests.output">
        <arg value="shell" />
        <arg value="am" />
        <arg value="instrument" />
        <arg value="-w" />
        <arg value="-e" />
        <arg value="coverage" />
        <arg value="@{emma.enabled}" />
        <arg value="${manifest.package}/${test.runner}" />
    </exec>
    <echo message="${tests.output}"/>
    <fail message="Tests failed!!!">
         <condition>
             <contains string="${tests.output}" substring="FAILURES" />
         </condition>
    </fail>
</target>

Thats better then my solution, I didnt like to amend the sdk xml — kirhgoff, Jan 10 '12 at 03:34
Could you please give more details? You appended that in the build.xml of the test project? — Raphael Oliveira, Mar 18 '13 at 17:33
this was long time ago, and yes, I think I added that to build.xml. — dongshengcn, Mar 18 '13 at 18:00

score 3 · Answer 3 · answered Jan 23 '11 at 02:38

Also was looking for some kind of standard solution for this. I wonder how do android guys develop, or they dont use teamcity and continuous integration? heard hudson has some plugin for android but I dont like hudson. anyway here is quick and dirty solution

replace contents in android-sdk-windows\tools\ant\test_rules.xml with:

        <attribute name="emma.enabled" default="false" />
    <element name="extra-instrument-args" optional="yes" />
    <sequential>
        <echo>Running tests ...</echo>
        <exec executable="${adb}" failonerror="true" outputproperty="tests.output">
            <arg line="${adb.device.arg}" />
            <arg value="shell" />
            <arg value="am" />
               <arg value="instrument" />
            <arg value="-w" />
            <arg value="-e" />
               <arg value="coverage" />
               <arg value="@{emma.enabled}" />
            <extra-instrument-args />
            <arg value="${manifest.package}/${test.runner}" />
    </exec>
    <echo message="${tests.output}"/>
    <fail message="Tests failed!!!">
            <condition>
                    <contains string="${tests.output}" substring="FAILURES" />
            </condition>
    </fail>
    </sequential>

there are two drawbacks 1) you dont see test output while tests are running until they failed (and the output is crippled somewhow) 2) its better to override this macro in your project

these days (as of SDK r15) the ant target is called "run-tests-helper" but the above workaround still works fine. And yes put the modified macro into your build.xml rather than altering the original in the android sdk. — Maks, Jan 10 '12 at 03:40

score 2 · Answer 4 · answered Mar 09 '11 at 10:38

another option would of course be to ditch Ant in favor of Maven or Gradle. Both have Android plug-ins that properly fail the build when there are test failures.

Maven: http://code.google.com/p/maven-android-plugin/

Gradle: http://code.google.com/p/gradle-android-plugin/

running instrumentation tests has just been added to the Gradle Android plug-in, and is waiting to be merged back into the master repository, so there should be another release soon.

score -3 · Accepted Answer · answered Jun 03 '10 at 15:04

-3

The ant JUnit task defaults to running all the tests. There are two solutions to this.

Easiest solution is to set the haltonerror property to true and the build will fail at the first test failure.

Slightly more involved (and my preference) is to set the failureProperty so that all the tests run. This lets you know how many tests fail instead of only the first test that fails. This requires more ant work because you need to add a line after your junit tests like this:

<fail message="tests failed" if="failureProperty"/>

answered Jun 03 '10 at 15:04

Alex B

24,678
14
64
87

how do you make this work? do you have to change android own build target? – dongshengcn Apr 20 '11 at 14:30
This is not helpful in this case as Androids ant test target actually uses the exec task to run the junit tests on a emulator/device using the adb tool. – Maks Sep 12 '12 at 05:48

Failing Android JUnit tests, not breaking my Ant script like I expect?

5 Answers5