Instrumentation tested app crashes due to Chrome auto-update on Firebase Test Lab

Question

While running instrumentation tests on Google's Firebase Test Lab, the tested app is crashing due to Google Chrome being updated in the background. The tested app indeed has some WebViews but they are not displayed during the test run.

The video recording distinctively shows apps being downloaded and installed (animated download notification) by the Play Store.

Relevant logcat:

03-19 22:52:52.450: I/ActivityManager(1154): Force stopping com.android.chrome appid=10086 user=-1: installPackageLI
03-19 22:52:52.450: I/ActivityManager(1154): Killing 31128:com.google.android.googlequicksearchbox:search/u0a54 (adj 500): stop com.android.chrome
03-19 22:52:52.454: I/ActivityManager(1154): Killing 15064:com.MYAPP/u0a159 (adj 0): stop com.android.chrome

Above it, I can clearly see the updates being downloaded, validated etc.

We've also tried running gcloud with the --no-auto-google-login flag with no success: apps are still being downloaded and installed.

Anyone else experienced this or has any suggestions?

Answer 1

To have better control over our instrumentation tests, we've ended up writing ChromeDisabler which will run every instrumentation test session, go into phone's Chrome settings and uninstall and disable Chrome, hence preventing it from auto-updating.

Ideally, Google should disable auto-updates for Firebase Test Lab images.

Usage:

class CustomTestRunner : AndroidJUnitRunner() {

    override fun onStart() {
        ChromeDisabler.disable()
        super.onStart()
    }
}

ChromeDisabler:

object ChromeDisabler {
    private const val CHROME_APP_PACKAGE_NAME = "com.android.chrome"
    private const val APPLICATION_DETAILS_SETTINGS_APP_PACKAGE_NAME = Settings.ACTION_APPLICATION_DETAILS_SETTINGS
    private const val WAIT_TIMEOUT_MILLIS = 5000L

    private val device = UiDevice.getInstance(InstrumentationRegistry.getInstrumentation())
    private val disableButton = device.findObject(UiSelector().textStartsWith("disable"))
    private val uninstallButton = device.findObject(UiSelector().textStartsWith("uninstall"))

    fun disable() {
        device.pressHome()

        launchSettingsActivityForChrome()

        device.wait(Until.hasObject(By.pkg(APPLICATION_DETAILS_SETTINGS_APP_PACKAGE_NAME).depth(0)), WAIT_TIMEOUT_MILLIS)

        when {
            disableButton.exists() -> {
                Timber.e("Stock $CHROME_APP_PACKAGE_NAME found")
                disableApp(device)
            }

            uninstallButton.exists() -> {
                Timber.e("Non-stock $CHROME_APP_PACKAGE_NAME found")
                uninstallApp(device)
                disableApp(device)
            }

            device.findObject(UiSelector().textStartsWith("enable")).exists() -> {
                Timber.e("$CHROME_APP_PACKAGE_NAME is already disabled")
            }
        }
    }

    private fun launchSettingsActivityForChrome() {
        val intent = Intent().apply {
            action = Settings.ACTION_APPLICATION_DETAILS_SETTINGS
            data = Uri.fromParts("package", CHROME_APP_PACKAGE_NAME, null)
            addFlags(Intent.FLAG_ACTIVITY_NEW_TASK)
        }

        InstrumentationRegistry.getInstrumentation().targetContext.startActivity(intent)
    }

    private fun disableApp(device: UiDevice) {
        Timber.e("Attempting to disable $CHROME_APP_PACKAGE_NAME")

        try {
            disableButton.click()
            device.findObject(UiSelector().textStartsWith("disable app")).click()
            Timber.e("Successfully disabled $CHROME_APP_PACKAGE_NAME")
        } catch (exception: Exception) {
            Timber.e("Failed to disable $CHROME_APP_PACKAGE_NAME, cause: $exception")
        }
    }

    private fun uninstallApp(device: UiDevice) {
        Timber.e("Attempting to uninstall $CHROME_APP_PACKAGE_NAME")

        try {
            uninstallButton.click()
            device.findObject(UiSelector().textStartsWith("ok")).click()
            Timber.e("Successfully uninstalled $CHROME_APP_PACKAGE_NAME")
        } catch (exception: Exception) {
            Timber.e("Failed to uninstall $CHROME_APP_PACKAGE_NAME, cause: $exception")
        }
    }
}