Randomizing TestNG Invocations

Question

Similar to: Run TestNG tests in random order but different as this question does not include invocations.

TestNG randomizes the tests themselves but the invocations will all run in a row. For Example:

@Test(invocation=2) public void A{}
@Test(invocation=2) public void B{}

will either run AABB or BBAA but I was hoping to have them run in a truly random manner (ABAB, BABA, ect)

Is there anyway to accomplish this with TestNG, at the moment all I can think of is changing my tests to methods called by one test that controls the order but I hoping for something built in to TestNG that I overlooked.

score 0 · Accepted Answer · answered Feb 10 '18 at 15:47

You can build on the same concept of leveraging a IMethodInterceptor.

Here's how you do it. First you build a custom annotation that can be used to express the invocation count.

import java.lang.annotation.Retention;
import java.lang.annotation.Target;

import static java.lang.annotation.ElementType.METHOD;
import static java.lang.annotation.ElementType.TYPE;

@Retention(java.lang.annotation.RetentionPolicy.RUNTIME)
@Target({METHOD, TYPE})
public @interface InvocationCount {
    int value() default 1;
}

You now customize your method interceptor such that it looks up the custom annotation and based on it, it adds up more instances and then eventually randomizes the resultant collection.

import org.testng.IMethodInstance;
import org.testng.IMethodInterceptor;
import org.testng.ITestContext;

import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import java.util.Random;

public class LocalInterceptor implements IMethodInterceptor {

    @Override
    public List<IMethodInstance> intercept(List<IMethodInstance> methods, ITestContext context) {
        List<IMethodInstance> altered = new ArrayList<>();
        for (IMethodInstance method : methods) {
            InvocationCount count = method.getMethod().getConstructorOrMethod().getMethod().getAnnotation(InvocationCount.class);
            if (count == null) {
                altered.add(method);
            } else {
                if (count.value() > 1) {
                    for (int i = 0; i < count.value(); i++) {
                        altered.add(method);
                    }
                }
            }
        }
        long seed = System.nanoTime();
        Collections.shuffle(altered, new Random(seed));
        return altered;
    }
}

Your test would now look like below

import org.testng.annotations.Listeners;
import org.testng.annotations.Test;

@Listeners(LocalInterceptor.class)
public class SampleTestClass {

    @InvocationCount(2)
    @Test
    public void A() {
        System.err.println("A()");
    }

    @InvocationCount(3)
    @Test
    public void B() {
        System.err.println("B()");
    }
}

Here's a sample output (There were some StringIndexOutOfBoundsException exceptions from the reporters, but I think that's not related to this)

...
... TestNG 6.14.2 by Cédric Beust (cedric@beust.com)
...
B()
A()
B()
B()
A()
PASSED: B
PASSED: A
PASSED: B
PASSED: B
PASSED: A

===============================================
    48715503_test
    Tests run: 5, Failures: 0, Skips: 0
===============================================

===============================================
48715503_Suite
Total tests run: 5, Failures: 0, Skips: 0
===============================================

Randomizing TestNG Invocations

1 Answers1