Instrumentation Tests: define a test as mandatory for the following

Question

I have an Android instrumentation test which tests my ServerApi and multiple other instrumentations which test e.g. my Activities.

One of the reasons the ServerApi test can fail is (sadly) a broken wifi. In this case, my whole test suite takes ages to complete because all the activities dealing with network access have to run into timeouts before failing.

How can I define the ServerApiTest to run in the beginning and if it fails, cancelling all other following tests?

public class ServerApiTest extends ApplicationTestCase<Application> {
    //...

    public ServerApiTest() {
        super(Application.class);
    }

    @Override
    public void setUp() throws Exception {
        Log.i(TAG, SERVER_KEYWORD + ": prepare");
        super.setUp();
        _context = getContext();
    }

    @Override
    public void tearDown() throws Exception {
        Log.i(TAG, SERVER_KEYWORD + ": cleanUp");
        super.tearDown();
    }

    public void testPing() throws InterruptedException, ExecutionException,
                                  URISyntaxException {
        //...
    }

As an example an excerpt of one of the Activity tests:

@RunWith(AndroidJUnit4.class)
@MediumTest
public class RecordingActivityTest {
    private static final String TAG = RecordingActivityTest.class.getSimpleName();

    @Rule
    public ActivityTestRule<RecordingActivity> _activityRule
        = new ActivityTestRule<>(RecordingActivity.class);


    @Test
    public void testOnCreate() {
        onView(withId(R.id.lytRecordingButtons)).check(
               matches(withEffectiveVisibility(ViewMatchers.Visibility.VISIBLE)));
    }
    //...

Interestingly, I have observed in the test reports that on some (rare and yet to be explained) occasions if some test fails, the test suite indeed seems to abort.

See the screenshots: the RecordingActivityTest failed and the number of tests is reported as 25. But with no changes in the tests (only fixing the bug which caused RecordingActivityTest to fail), the next build correctly showed all 43 tests again - coincidentally now with the tests failing due to a broken WiFi.

First test run which does not count all tests: Following test run which shows all tests again:

Whatever caused the tests not to be executed, could I use this mechanism for what I want to achieve?

score 1 · Answer 1 · answered Jul 14 '17 at 15:00

1

If a stable Wifi is needed to run the tests, and you want to skip running them otherwise, you could implement a custom TestRule where you do the Wifi check in a custom Statement within that rule. In the evaluate method of that statement, you then call Assume.assumeTrue("Skip test due to poor Wifi connection", methodThatReturnsTrueIfWifiIsUp()). You could then add this rule to each test class that needs Wifi, and possibly store a static connection state if the network lookup takes a long time to complete and you don't want to do it for each test class.

answered Jul 14 '17 at 15:00

fejd

2,545
1
16
39

This would be okay but the test report would be harder to understand because a lot of tests fail and one would have to infer that it is all because of the same reason (by looking deeper in the reports). If only one test would fail and stop the execution of the following, this would be clear immediately – PhilLab Jul 21 '17 at 14:21
@PhilLab This could be helpful in that case https://stackoverflow.com/questions/10036894/stopping-junit-suite-if-particular-test-fails. – fejd Jul 23 '17 at 22:33

Instrumentation Tests: define a test as mandatory for the following

1 Answers1