How can I build a configurable JUnit4 test suite?

Question

Guava has an extensive set of tests for collection implementations written in JUnit3 that look like:

/*
 * Copyright (C) 2008 The Guava Authors
 *
 * Licensed under the Apache License, Version 2.0 (the "License");
 * you may not use this file except in compliance with the License.
 * You may obtain a copy of the License at
 *
 * http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
public class CollectionRemoveTester<E> extends AbstractTester<E> {

  @CollectionFeature.Require(SUPPORTS_REMOVE)
  @CollectionSize.Require(absent = ZERO)
  public void testRemove_present() {
     ...
  }
}

and then different collections are tested by using TestSuiteBuilders that pass in a set of features and generators for the collection type, and a heavily reflective framework identifies the set of test methods to run.

I would like to build something similar in JUnit4, but it's not clear to me how to go about it: building my own Runner? Theories? My best guess so far is to write something like

abstract class AbstractCollectionTest<E> {
   abstract Collection<E> create(E... elements);
   abstract Set<Feature> features();

   @Test
   public void removePresentValue() {
      Assume.assumeTrue(features().contains(SUPPORTS_REMOVE));
      ...
   }
}

@RunWith(JUnit4.class)
class MyListImplTest<E> extends AbstractCollectionTest<E> {
  // fill in abstract methods
}

The general question is something like: how, in JUnit4, might I build a suite of tests for an interface type, and then apply those tests to individual implementations?

Answer 1

In Junit you can use categories. For example this suite will execute al test from the AllTestSuite annotated as integration:

import org.junit.experimental.categories.Categories;
import org.junit.experimental.categories.Categories.IncludeCategory;
import org.junit.runner.RunWith;
import org.junit.runners.Suite;

@RunWith(Categories.class)
@IncludeCategory(Integration.class)
@Suite.SuiteClasses ({AllTestsSuite.class} )
public class IntegrationTestSuite {}

You can also use @ExcludeCategory. This is usefull to remove slow tests. Categories classes are just plain old Java classes or interfaces. For example:

public interface Integration{}
public interface Performance{}
public interface Slow{}
public interface Database{}

You only need to anotate your tests acordingly:

@Category(Integration.class)
public class MyTest{

   @Test
   public void myTest__expectedResults(){
   [...]

One test might have more than one category like this:

   @Category({Integration.class,Database.class})  
   public class MyDAOTest{

For simplicity I usually create a Suite with all classes in the test folder using google toolbox:

import org.junit.runner.RunWith;

import com.googlecode.junittoolbox.ParallelSuite;
import com.googlecode.junittoolbox.SuiteClasses;

@RunWith(ParallelSuite.class)
@SuiteClasses({"**/**.class",           //All classes
             enter code here  "!**/**Suite.class" })    //Excepts suites
public class AllTestsSuite {}

This works incluiding in AllTestSuite all classes in the same folder and subfolders even if they don't have the _Test sufix. But won't be able to see test that are not in the same folder or subfolders. junit-toolbox is available in Maven with:

<dependency>
    <groupId>com.googlecode.junit-toolbox</groupId>
    <artifactId>junit-toolbox</artifactId>
    <version>2.2</version>
</dependency>

Now you only need to execute the Suite that suits your needs :)

UPDATE: In Spring there is the @IfProfileValue annotation that allows you to execute test conditionally like:

@IfProfileValue(name="test-groups", values={"unit-tests", "integration-tests"})
@Test
public void testProcessWhichRunsForUnitOrIntegrationTestGroups() {

For more information see Spring JUnit Testing Annotations

Answer 2

Regarding whether or not to build your own Runner... I think you shouldn't immediately try to build your own but parametrerize your unit tests instead.

One option is to annotate your class with @RunWith(Parameterized.class) and insert a static method annotated with @Parameters that will be used for the actual parameterization using the constructor of the JUnit test. Below an example I shamelessly took from https://github.com/junit-team/junit/wiki/Parameterized-tests:

@RunWith(Parameterized.class)
public class FibonacciTest {
    @Parameters
    public static Collection<Object[]> data() {
        return Arrays.asList(new Object[][] {     
                 { 0, 0 }, { 1, 1 }, { 2, 1 }, { 3, 2 }, { 4, 3 }, { 5, 5 }, { 6, 8 }  
           });
    }

    private int fInput;

    private int fExpected;

    public FibonacciTest(int input, int expected) {
        fInput= input;
        fExpected= expected;
    }

    @Test
    public void test() {
        assertEquals(fExpected, Fibonacci.compute(fInput));
    }
}

This will have all your test methods use the same parameters as they will be usually assigned to corresponding fields in the JUnit class. The key will be the instantiation of the different implementations in this static method (Dagger, Guice, factories, whatever). They will be then automatically be then passed to the constructor and you will be responsible for assigning them to the fields you are going to use in the testing methods. As you see, instead of using the example's array of integers, just place instances of your implementation inside. For more information look at the link above.

The second option is to use Zohhak with the annotation @RunWith(ZohhakRunner.class) from https://github.com/piotrturski/zohhak. This will allow you to parameterize your unit tests per method instead of per class. This could be trickier with the factory instantiation, but it can be made quite elegantly with a bit of work. Example taken from the Zohhak site:

@TestWith({
    "clerk,      45'000 USD, GOLD",
    "supervisor, 60'000 GBP, PLATINUM"
})
public void canAcceptDebit(Employee employee, Money money, ClientType clientType) {
    assertTrue(   employee.canAcceptDebit(money, clientType)   );
}

I would start with the first approach and if you hit alimit, move to the second. Cheers and good luck.

Answer 3

You can use JUnit rules to conditionally ignore tests (they will end up as skipped in maven report, but there might be a way to fiddle around this that I don't know about).

This is based on rule in this article. I changed the rule a bit, see here.

public abstract class AbstractCollectionTest {

@Rule
public ConditionalSupportRule rule = new ConditionalSupportRule();

private Collection<String> collection;
private Set<Class<? extends Feature>> features;

public AbstractCollectionTest(Collection<String> collection,
                              Class<? extends Feature> ... features) {
    this.collection = collection;

    this.features = new HashSet<>();
    for (Class<? extends Feature> feature : features) {
        this.features.add(feature);
    }
}

@Test
@ConditionalSupport(condition = SupportsRemove.class)
public void remove() throws Exception {

    // ...
    System.out.println("test run");
}

private interface Feature {}

public class SupportsRemove implements RunCondition, Feature {

    public SupportsRemove() {
    }

    @Override
    public boolean isSatisfied() {
        return features.contains(SupportsRemove.class);
    }
}

Example test for array list:

public class ArrayListTest extends AbstractCollectionTest {

    public ArrayListTest() {
        super(new ArrayList<>(), SupportsRemove.class);
    }

}

Some list that doesn't support remove:

public class UnmodifiableListTest extends AbstractCollectionTest {

    public UnmodifiableListTest() {
        super(Collections.unmodifiableList(new ArrayList<>()));
    }
}