7

I am working on a project where in we are trying to use the ANDROID TEST ORCHESTRATOR for it's obvious benefits of isolating crashes. But while executing the test suite, it appears to me that as the orchestrator initiates a new process for every test case, the test coverage report for the suite execution is always showing incomplete data (mostly the data for the last test case present in the test suite).

So I was wondering that is there a way to overcome this problem and generate a jacoco code coverage report for all instrumented tests existing in the test suite.

Shredder
  • 150
  • 1
  • 1
  • 15
  • I am still searching for answers to this, any idea is also invited, does not need to provide an exact solution. – Shredder Jan 08 '18 at 14:17
  • 1
    I found an issue similar to this or the same where I needed to remove orchestration to get androidTest code coverage. Still haven't resolved it. You can see our gradle here: https://stackoverflow.com/questions/48427286/unit-test-and-connected-test-jacoco-execution-data-format-differ-causes-sonarqu We too still haven't worked out the orchestration bit and would like to know as well. Here is the only related tracker issue I could find: https://issuetracker.google.com/issues/71989799 – JTT Jan 31 '18 at 20:51
  • 1
    Logged this bug: https://issuetracker.google.com/issues/72758547 – JTT Feb 01 '18 at 14:13
  • @Shredder What did you end up doing ? – Islam Salah Feb 08 '18 at 15:29
  • 1
    @IslamSalah only observation from my end is that for each and every test case a separate coverage file is being generated, so currently i am building a system to keep on listening to the path (where the coverage file gets dumped), and then collate all the coverage reports into one. But hope google comes out with a more optimised solution soon. Let me know if you are trying something else as well. – Shredder Feb 09 '18 at 10:33
  • Google has made a beta release of orchestrator to fix the issue at their end (more info at https://developer.android.com/training/testing/release-notes), but still Gradle and Android Studio teams need to pickup and implement the fix at their end to provide a completely working solution. – Shredder May 14 '18 at 12:55

2 Answers2

3

If you're using the android test orchestrator this is the problem. There is an open bug report: https://issuetracker.google.com/issues/72758547

The only solution that I know of is to disable the android test orchestration until a fix is released.

In my Android Java build.gradle I had to comment out the test orchestrator like so:

android {
...
    testOptions {
        // temporarily disable the orchestrator as this breaks coverage: https://issuetracker.google.com/issues/72758547
        //execution 'ANDROID_TEST_ORCHESTRATOR'
    ...
    }
}
Inti
  • 3,443
  • 1
  • 29
  • 34
  • Thanks for pointing out the obvious, but I am trying to find a way to overcome the bug with some workaround as the app in test at my end does not persist without crashing off while the complete test suite execution. – Shredder Feb 24 '18 at 18:38
  • Whops - didn't realise you already knew about the bug report! Might still help someone who like me just wanted their coverage report to start working again :) – Inti Feb 26 '18 at 13:53
  • Yeah that's true :) Thanks anyway, let me know if you come across something more about it. – Shredder Feb 27 '18 at 02:52
-3

The issue mentioned above is now fixed and the latest gradle version is working fine for all android devices.

Shredder
  • 150
  • 1
  • 1
  • 15