Why are some tests not executed in my Gradle build?

Question

I have inherited a project with about 1000 tests. Until recently, all of them have been executed when I entered the gradle clean test command.

From one day to another some of the tests stopped being included in the test statistic (the number of passed, failed and ignored tests that Gradle and Idea output at the end of execution of all tests). The code is still there and the tests were not ignored. Nor was any of the build scripts modified. When I run the tests in IntelliJ Idea they are executed (i. e. there are no compiler errors that could have explained why the tests don't run).

I believe that the tests are executed, but they results are not included in the overall statistics for the following reason. I executed the command gradle --debug --rerun-tasks clean test > 2017_11_24_gradle.log. Then I looked for the occurrences of one of the missing tests, com.mycompany.comm.CommApplicationTests in the resulting 2017_11_24_gradle.log file (grep "com.mycompany.comm.CommApplicationTests" 2017_11_24_gradle.log).

This is grep's output:

17:00:54.861 [QUIET] [system.out] 17:00:54.860 [DEBUG] [org.gradle.api.internal.tasks.testing.junit.JUnitTestClassProcessor] Executing test class com.mycompany.comm.CommApplicationTests
17:00:54.867 [DEBUG] [TestEventLogger] com.mycompany.comm.CommApplicationTests STARTED
17:00:55.169 [DEBUG] [TestEventLogger] com.mycompany.comm.CommApplicationTests > validateProtocol STARTED
17:00:55.875 [DEBUG] [TestEventLogger] com.mycompany.comm.CommApplicationTests > validateProtocol STANDARD_OUT
17:00:57.272 [DEBUG] [TestEventLogger] com.mycompany.comm.CommApplicationTests > validateProtocol PASSED
17:00:57.273 [DEBUG] [TestEventLogger] com.mycompany.comm.CommApplicationTests > contextLoads STARTED
17:00:57.273 [DEBUG] [TestEventLogger] com.mycompany.comm.CommApplicationTests > contextLoads PASSED
17:00:57.274 [DEBUG] [TestEventLogger] com.mycompany.comm.CommApplicationTests PASSED

From this I conclude that the tests are executed.

The only suspicious thing which may or may not account for the non-inclusion of some test results is this:

16:37:47.251 [ERROR] [system.err] *** java.lang.instrument ASSERTION FAILED ***: "!errorOutstanding" with message transform method call failed at JPLISAgent.c line: 844
16:37:47.343 [ERROR] [system.err] *** java.lang.instrument ASSERTION FAILED ***: "!errorOutstanding" with message transform method call failed at JPLISAgent.c line: 844
16:37:47.343 [ERROR] [system.err] *** java.lang.instrument ASSERTION FAILED ***: "!errorOutstanding" with message transform method call failed at JPLISAgent.c line: 844
16:37:47.346 [ERROR] [system.err] *** java.lang.instrument ASSERTION FAILED ***: "!errorOutstanding" with message transform method call failed at JPLISAgent.c line: 844
16:37:47.346 [ERROR] [system.err] *** java.lang.instrument ASSERTION FAILED ***: "!errorOutstanding" with message transform method call failed at JPLISAgent.c line: 844
16:37:47.347 [ERROR] [system.err] *** java.lang.instrument ASSERTION FAILED ***: "!errorOutstanding" with message transform method call failed at JPLISAgent.c line: 844

These messages are output to System.err during the execution of gradle --debug --rerun-tasks clean test.

How can I fix or work around the erorr (i. e. make sure that all executed tests are reflected in the end statistics (number of passed, failed, and ignored tests) that Gradle prints at the end)?

Update 1 (27.11.2017 10:49 MSK):

It looks like the error occurs in the code fragment below (comment // Line 844):

if ( !errorOutstanding ) {
    jplis_assert(agent->mInstrumentationImpl != NULL);
    jplis_assert(agent->mTransform != NULL);
    transformedBufferObject = (*jnienv)->CallObjectMethod(
                                        jnienv,
                                        agent->mInstrumentationImpl,
                                        agent->mTransform,
                                        loaderObject,
                                        classNameStringObject,
                                        classBeingRedefined,
                                        protectionDomain,
                                        classFileBufferObject,
                                        is_retransformer);
    errorOutstanding = checkForAndClearThrowable(jnienv);
    jplis_assert_msg(!errorOutstanding, "transform method call failed"); // Line 844
}

Update 2 (27.11.2017 16:51 MSK): This error does not occur on Mac (Gradle reports correct numbers of total, passed and failed tests).

Update 3 (29.11.2017 12:48 MSK):

I'm using Java 1.8.0_152 and Gradle 2.14. I cannot upgrade the Gradle version due to the customer requirements.

java -version
java version "1.8.0_152"
Java(TM) SE Runtime Environment (build 1.8.0_152-b16)
Java HotSpot(TM) 64-Bit Server VM (build 25.152-b16, mixed mode)


gradle -version

------------------------------------------------------------
Gradle 2.14
------------------------------------------------------------

Build time:   2016-06-14 07:16:37 UTC
Revision:     cba5fea19f1e0c6a00cc904828a6ec4e11739abc

Groovy:       2.4.4
Ant:          Apache Ant(TM) version 1.9.6 compiled on June 29 2015
JVM:          1.8.0_152 (Oracle Corporation 25.152-b16)
OS:           Windows 10 10.0 amd64

given your warning comes from the JDK, it might be more meaningful to say which Java version you are using on the systems with / without the error. Also the gradle version might be meaningful. And finally, the gradle forums might be a better place to get support about this. — tkruse, Nov 29 '17 at 01:43

score 0 · Answer 1 · answered Nov 28 '17 at 14:23

0

It is hard to tell without seeing the test code. However, what exactly makes you think that the "results are not included in the overall statistics"? I do not see any notion of that? Did the reported number of tests decrease?

Or is it just the error messages? These seem to be "only" error messages reported to the standard error output. Such errors do not necessarily mean the test failed, they are just "warnings" - in the sense nothing is done except for writing them to the error output, which can be seen for instance here.

So, my impression is that the tests indeed passed, they just printed a (warning) error message. The fact they behave slightly differently on Mac may be caused by many circumstances.

answered Nov 28 '17 at 14:23

Marwin

2,655
1
19
17

Re "what exactly makes you think that the "results are not included in the overall statistics"?": Several things. First, some test classes do not appear in the final report (if I execute the same Gradle task in Idea, I can export the test results to HTML). Those "missing" classes do exist and have no errors (I can execute these tests in Idea). 2) In the Gradle log file these test classes appear in messages like `X STARTED` and `X PASSED` (X - name of the test class). When I compare the exported reports between two days I see that some tests were there one day, and disappeared the next one. – Glory to Russia Nov 28 '17 at 15:15
I have to add that this seems to be a phantom bug because sometimes the correct results are being reported. – Glory to Russia Nov 28 '17 at 15:15
Sometimes? Isn't that some issue of gradle task dependencies? For instance, some classes may not compile but are executed when already compiled. Do you always use "clean" before "test"? It is really difficult to say without seeing that. – Marwin Nov 28 '17 at 16:05
Re " Isn't that some issue of gradle task dependencies?": If there was a problem with dependencies, the build would always fail. – Glory to Russia Nov 28 '17 at 17:23
Not necessarily. Back to the issue, though. You're claiming that some tests are logged as STARTED and PASSED by TestEventLogger and yet *the same tests* do not appear in the test report/statistics *in the same run*. Correct? That would be really weird and I'm afraid it cannot be resolved without seeing a (non)working example. – Marwin Nov 29 '17 at 10:41

Why are some tests not executed in my Gradle build?

1 Answers1