Should a TDD test always fail first?

Question

As a followon to the discussion in the comments of this answer, should a TDD test always be made fail first?

Consider the following example. If I am writing an implementation of LinkedHashSet and one test tests that after inserting a duplicate, the original is in the same iteration order as before the insert, I might want to add a separate test that the duplicate is not in the set at all.

The first test will be observed to fail first, and then implemented.

The problem is that it is quite likely that the implementation to make the first test pass used a different set implementation to store the data, so just as a side effect the second test already passes.

I would think that the main purpose of seeing the test fail is to ensure that the test is a good test (many times I've written a test I thought would fail but didn't because the test was written wrong). But if you are confident that the test you write does indeed test something, isn't it valuable to have to ensure that you don't break that behavior later?

Answer 1

Of course it's valuable, because then it is a useful regression test. In my opinion, regression tests are more important than testing newly developed code.

To say that they must always fail first is taking a rule beyond practicality.

Answer 2

Yes, TDD tests must fail before they turn green (work). Otherwise you do not know if you have a valid test.

Answer 3

TDD for me is more of a design tool, not an afterthought. So there is no other way, the test will fail simply because there is no code to make it pass yet, only after I create it that the test can ever pass.

Answer 4

I think the point of "failing first" is to avoid kidding yourself that a test worked. If you have a set of tests checking the same method with different parameters, one (or more) of them is likely to pass from the start. Consider this example:

public String doFoo(int param) {
    //TODO implement me
    return null;
}

The tests would be something like:

public void testDoFoo_matches() {
    assertEquals("Geoff Hurst", createBar().doFoo(1966));
}

public void testDoFoo_validNoMatch() {
    assertEquals("no match", createBar().doFoo(1));
}

public void testDoFoo_outOfRange() {
    assertEquals(null, createBar().doFoo(-1));
}
public void testDoFoo_tryAgain() {
    assertEquals("try again", createBar().doFoo(0));
}

One of those tests will pass, but clearly the others won't, so you have to implement the code properly for the set of tests to pass. I think that is the true requirement. The spirit of the rule is to ensure you have thought about the expected outcome before you start hacking.

Answer 5

What you're actually asking is how you can test the test to verify that it is a valid one and it tests what you intend.

Making it fail at first is an ok option, but note that even if it fails when you plan it to fail and succeeds after you refactor the code to make it succeed, that still doesn't mean that your test actually tested what you wanted... Of course you can write some other classes which behave differently to test your test... But that's actually a test which tests your original test - How do you know that the new test is valid? :-)

So making a test fail first is a good idea but it still isn't foolproof.

Answer 6

IMHO, the importance of failing first is to make sure that the test you created doesn't have a flaw. You could, for instance, forget the Assert in your test, and you'd maybe never know that.

A similar case occurs when you're doing boundary tests, you've already built the code that covers it, but it is recommended to test that.

I think that's not a big problem for your test not to fail, but you have to make sure it is indeed testing what it should (debugging, maybe).