TestNG vs Spock for Automation

Question

We are looking at implementing a test framework and curious as to which framework to use. We are choosing between TestNG and Spock. This is going to be a UI Automation framework so it should handle as little mock data as possible. Our codebase will be comprised of Geb (Groovy).

That being said, there are 3 advantages that Spock holds over TestNG:

Detailed information Spock's runtime collects a wealth of information, and presents it to you when needed. Condition not satisfied:

max(a, b) == c
|   |  |  |  |
3   1  3  |  2
          false

Beautiful language Express your thoughts in a beautiful and highly expressive specification language.

def "subscribers receive published events at least once"() {
    when: publisher.send(event)
    then: (1.._) * subscriber.receive(event)
    where: event << ["started", "paused", "stopped"]
}

Extensible for everyone @Transaction? @SpringBean? @DeployApp? With Spock's interception-based extension mechanism, you can easily create your own extensions.

Does anyone have any input on why one may be better than the other?

Are there downfalls to either?

Is there a way to create the "beautiful language" in reports for TestNG? Essentially can I create my own tags and have a program that parses for them? Or is there already a third party library to add?

Answer 1

For UI automation, take a look at Geb before trying to build a new framework. http://www.gebish.org

Geb will work with both spock and TestNG.

In general, I've had really good experience with spock, but can't speak to TestNG.

I do wish spock had BDD style reports like Cucumber, and property style testing like ScalaCheck, but as far as straightforward TDD style testing, spock is expressive, easy to use, feature rich, and really well designed. Compared to things like JUnit, spock is a joy - very little ceremony needed.

Answer 2

I am using TestNG for a few years, a few projects and I am quite happy with it. Answering specific points from your question

Beautiful language

Test method name must be valid Java method name. That means no spaces. I start my methods with should, so I would name your method subscribers receive published events at least once as shouldSubscrbersReceiveEvents. I believe this shorter name is describing intention good enough. Camel case names are not as readable as proper sentences, but I see no other choice.

Given-when-then sections

TestNG has no build-in feature to support that. I am doing that manually, using comments. There is no framework support for features like:

• parameters comming from where: sections. You may use @DataProviders for that.
• assertions for statements in then: section. You must write assertions (assertThat) on your own.

My sample test:

 public void shouldUnmarshalDataHeader() throws IOException {
        //given DataHeader unmarshalled from InputStream
        DataHeader actual = sut.parseDataHeader(stringToInputStream(BINARY_DATA_HEADER));

        //when we create DataHeader using java API
        DataHeader expected = DataHeaderBuilder
            .sampleInstance(sut.dateTimeProvider.getLocalDateTime());

        //then those two objects are equal
        Assertions.assertThat(actual).isEqualTo(expected);
    }

Detailed information

This feature belongs to assertion library, not the test library. TestNG has build-in assertion library but you may use any other library. For me https://joel-costigliola.github.io/assertj/ works best. Assertion messages are better than ones from plain testNG or JUnit. AssertJ offers fluent API. There is very little learning curve.

When you have import static org.assertj.core.api.Assertions.*;, just type assertThat(something). and code complete from IDE will guide you. The better assertions you use, the more readable your test will be and better error messages you will get.

Extensible

Spring has build-in support for TestNG. Your test must extend AbstractTestNGSpringContextTests or AbstractTransactionalTestNGSpringContextTests. Then you are able to load any Spring Context, @Autowire spring bean into your tests, use transactions.

See http://docs.spring.io/spring/docs/current/spring-framework-reference/html/integration-testing.html#testcontext-support-classes-testng for more details.

Summary

For me TestNG is working well. You are able to test everything that you are able to test using Spock. Because TestNG is pure Java, DSL for writing the tests will not be as feature-rich as Groovy in Spock. But I believe that TestNG+AssertJ has more smooth learing curve than Spock (for Java developer). If somebody is interested in learing Groovy (maybe you want to use Groovy for production code too?) then Spock is fine choice, worth initial investment in learing.

For learning TestNG I highly recommend 'Practical Unit Testing with TestNG and Mockito' (http://practicalunittesting.com/) and 'Next Generation Java Testing: TestNG and Advanced Concepts' (http://testng.org/doc/book.html).

Update Oct 2018

Above post was written in 2016. Currently we have Junit5. Junit5 closes the gap between junit4 and TestNG. There are a few places where Junit5 is more feature-rich than testNG (example: better support for parametrized test methods).

I have used jUnit5 in few comercial projects and I am happy with it.

Junit5 is still pure java, so no language-level support for given-when-then-sections or assertions as good as in spock. I still use AssertJ and a few other companion tools (example:https://github.com/awaitility/awaitility for testing asynchronous operations)