How can I use @FlakyTest annotation now?

Question

I try to run flaky test with espresso framework (and with Junit4) on Android Studio.

I want to set how many times it should to repeat. Before I can use

@FlakyTest(tolerance=5)

// (5 is number for repeat, for example)

But this annotation was deprecated in API level 24. - (link on android.developers.com)

Now is availible new @FlakyTest annotation - without tolerance variable. (link on android.developers.com)

I need to set how many times test can be repeated, but don't know how to do it. Any idea?

Seems like the best you can do with new instrumentation runner is explicitly start all `FlakyTest`-annotated tests. — azizbekian, Aug 11 '17 at 14:21
then what is the purpose of FlakyTest if we cannot put tolerance ? — j2emanue, Dec 18 '17 at 06:44

andrei_zaitcev · Answer 1 · 2018-06-16T08:03:01.233

This annotation has been deprecated because the entire testing framework was replaced by a new one. Thus, the annotation has been also deprecated in favor of a new one.

Unfortunately, comparing to the old annotation this one can not be used to re-run a failed test. That makes it less useful for a practical purpose.

However, you can still use it for something useful. As the documentation says when running tests you can filter out those that are flaky. To do so you need to adjust a build script:

android {
    defaultConfig {
        testInstrumentationRunnerArgument "notAnnotation", "android.support.test.filters.FlakyTest"
    }
}

More about options can be found here.

score 4 · Answer 2 · answered Jun 13 '19 at 13:59

I found this online https://gist.github.com/abyx/897229#gistcomment-2851489

You create your own test rule

class RetryTestRule(val retryCount: Int = 3) : TestRule {

    private val TAG = RetryTestRule::class.java.simpleName

    override fun apply(base: Statement, description: Description): Statement {
        return statement(base, description)
    }

    private fun statement(base: Statement, description: Description): Statement {
        return object : Statement() {

            override fun evaluate() {
                Log.e(TAG, "Evaluating ${description.methodName}")

                var caughtThrowable: Throwable? = null

                for (i in 0 until retryCount) {
                    try {
                        base.evaluate()
                        return
                    } catch (t: Throwable) {
                        caughtThrowable = t
                        Log.e(TAG, description.methodName + ": run " + (i + 1) + " failed")
                    }
                }

                Log.e(TAG, description.methodName + ": giving up after " + retryCount + " failures")
                if (caughtThrowable != null)
                    throw caughtThrowable
            }
        }
    }
}

and then add it to your test, like this

@Rule
@JvmField
val mRetryTestRule = RetryTestRule()

How can I use @FlakyTest annotation now?

2 Answers2