How can I make my JUnit tests run in random order?

Question

I have the classical structure for tests, I have a test suite of different suites like DatabaseTests, UnitTests etc. Sometimes those suites contains other suites like SlowDatabaseTests, FastDatabaseTests etc.

What I want is to randomize the running order of tests so I will make sure they are not dependent to each other. Randomization should be at every level, like suite should shuffle test class order, and test class should shuffle test method order.

If it is possible to do this in Eclipse that will be the best.

Answer 1

You do have a Sortable but I can't see how you would use it.

You could extend BlockJUnit4ClassRunner and have computeTestMethods() return a randomized copy of super.computeTestMethods(). Then use the @RunWith to set that as the runner to use.

e.g.

package com.stackoverflow.mlk;

import java.util.Collections;

import org.junit.runners.BlockJUnit4ClassRunner;
import org.junit.runners.model.InitializationError;

public class RandomBlockJUnit4ClassRunner extends BlockJUnit4ClassRunner {

    public RandomBlockJUnit4ClassRunner(Class<?> klass)
            throws InitializationError {
        super(klass);
    }

    protected java.util.List<org.junit.runners.model.FrameworkMethod> computeTestMethods() {
        java.util.List<org.junit.runners.model.FrameworkMethod> methods = super.computeTestMethods();
        Collections.shuffle(methods);
        return methods;
    }

}

Then

@RunWith(com.stackoverflow.mlk.RandomBlockJUnit4ClassRunner.class)
public class RandomOrder {
    @Test
    public void one() {
    }

    @Test
    public void two() {
    }

    @Test
    public void three() {
    }
}

Answer 2

https://github.com/KentBeck/junit/pull/386 introduces some orders but not RANDOM. Probably you do not really want this; tests should run deterministically. If you need to verify that different permutations of tests still pass, either test all permutations; or, if this would be impractically slow, introduce a “random” seed for shuffling that is determined by an environment variable or the like, so that you can reproduce any failures. http://hg.netbeans.org/main/file/66d9fb12e98f/nbjunit/src/org/netbeans/junit/MethodOrder.java gives an example of doing this for JUnit 3.

Answer 3

In general what you need to do is to write your own test runner and in the test runner class aggregate the methods and randomly run each test (make sure you don't run a test twice).

Read more about the test framework and how to write your own test runner here: http://www.ddj.com/architect/184415674

Answer 4

In JUnit 4.13, to run the tests within a test class in random order, write a small helper class:

import org.junit.runner.manipulation.Ordering;

import java.util.Random;

public class RandomOrder implements Ordering.Factory {
    @Override
    public Ordering create(Ordering.Context context) {
        long seed = new Random().nextLong();
        System.out.println("RandomOrder: seed = " + seed);
        return Ordering.shuffledBy(new Random(seed));
    }
}

Then, annotate your test class with:

@OrderWith(RandomOrder.class)

This way, the test methods of this one class are run in random order. Plus, if they unexpectedly fail, you know the random seed to repeat exactly this order.

I don't know though how to configure this for a whole project or a test suite.

Answer 5

I will make sure they are not dependent to each other

You should make sure that this is the case without relying on random execution order. What makes you fear that dependencies may exist?

Answer 6

This issue is open on JUnit GitHub since 2 years, and point out 2 independent issues:
- Tests depending on the execution order;
- Non repeatable tests.

Consider adressing the issue at the root, rather than trying to use the framework to do the job afterwards. Use setUp and tearDown method to guarantee isolation, and test at the smallest level.

Answer 7

Here is a solution with Gradle and JUnit 5.8.0

Step 1 : Ensure that you have latest JUnit version dependency.
Step 2 : Define the required properties under build.gradle test section

test {  
    useJUnitPlatform()
    systemProperties([
         //Random in method level
        'junit.jupiter.testmethod.order.default': 'org.junit.jupiter.api.MethodOrderer$Random', 
         // Random in class level
        'junit.jupiter.testclass.order.default' : 'org.junit.jupiter.api.ClassOrderer$Random',
        // Log configuration to see the seed
        'java.util.logging.config.file' : file('src/test/resources/logging.properties')
    ])
    //To print the JUnit logs in console
    testLogging {
        events "passed", "skipped", "failed", "standardOut", "standardError"
    }
}

Step 3: Define logging.properties under src/test/resources

.level=CONFIG
java.util.logging.ConsoleHandler.level=CONFIG
org.junit.jupiter.api.ClassOrderer$Random.handlers=java.util.logging.ConsoleHandler
org.junit.jupiter.api.MethodOrderer$Random.handlers=java.util.logging.ConsoleHandler

Step 4 : Run test. gradlew clean test

You can see the seed used for the random test in the console

CONFIG: ClassOrderer.Random default seed: 65423695211256721
CONFIG: MethodOrderer.Random default seed: 6542369521653287

In case of flaky test, you can reproduce it by configuring the same seed where the JUnit tests were failing

systemProperties([
'junit.jupiter.execution.class.order.random.seed' : '65423695211256721'
'junit.jupiter.execution.order.random.seed'       : '6542369521653287'
])

References : how-to-randomize-tests-in-junit , Random