Prove correctness of unit test

Question

I'm creating a graph framework for learning purposes. I'm using a TDD approach, so I'm writing a lot of unit tests. However, I'm still figuring out how to prove the correctness of my unit tests

For example, I have this class (not including the implementation, and I have simplified it)

public class SimpleGraph(){
 //Returns true on success
 public boolean addEdge(Vertex v1, Vertex v2) { ... }

 //Returns true on sucess
 public boolean addVertex(Vertex v1) { ... }
}

I also have created this unit tests

@Test
public void SimpleGraph_addVertex_noSelfLoopsAllowed(){
 SimpleGraph g = new SimpleGraph();
 Vertex v1 = new Vertex('Vertex 1');
 actual = g.addVertex(v1);
 boolean expected = false;
 boolean actual = g.addEdge(v1,v1);
 Assert.assertEquals(expected,actual);
}

Okay, awesome it works. There is only one crux here, I have proved that the functions work for this case only. However, in my graph theory courses, all I'm doing is proving theorems mathematically (induction, contradiction etc. etc.).

So I was wondering is there a way I can prove my unit tests mathematically for correctness? So is there a good practice for this. So we're testing the unit for correctness, instead of testing it for one certain outcome.

Answer 1

No. Unit tests don't attempt to prove correctness in the general case. They should test specific examples. The idea is to pick enough representative examples that if there is an error it will probably be found by one or more of the tests, but you can't be sure to catch all errors this way. For example if you were unit testing an add function you might test some positive numbers, some negative, some large numbers and some small, but using this approach alone you'd be lucky to find the case where this implementation doesn't work:

int add(int a, int b) {
    if (a == 1234567 && b == 2461357) { return 42; }
    return a + b;
}

You would however be able to spot this error by combining unit testing and code coverage. However even with 100% code coverage there can be logical errors which didn't get caught by any tests.

It is possible to prove code for correctness. It is called formal verification, but it's not what unit tests are for. It's also expensive to do for all but the most simple software so it is rarely done in practice.

Answer 2

Probably not. Unit tests approach the problem by exhaustive testing:

• You verify that your test works by writing the test before implementing the behavior.
• Then you see that the test fails.
• Then you implement the behavior to pass that test, and only that test. Never write code that is not needed to implement a test.

Answer 3

Really, what you're proving is that one case of your algorithm is working, eg you're proving that a subset of your execution paths are valid. Testing will never help you prove correctness in the strict mathematical sense (except for very simple cases). In the general case, this is impossible. Testing is a pragmatic approach to this problem where we try to show representative cases are correct (boundary values, values somewhere in the middle, etc.) and hope that that works.

Still, some tools such as findbugs etc. manage to give you conservative proof of some properties of your code.

If you would like formal proof of your stuff, there's always Coq, Agda and similar languages, but that's a hell of a stretch from writing a unit test :)

One great, simple introduction to testing vs proofs is Abstract Interpretation in a Nutshell Patrick Cousot.

Answer 4

My 2 cents. Look at it this way: you think you wrote a function that does something, but what you really did was writing a function that you think it does something. If you cannot write a mathematically proof of what the code does, you can as well treat the function as a hypothesis; you cannot be sure it will be always correct, but at least it is falsiable.

And that's why we write unit testing (note: just other functions, prone to have bugs, sigh), to try to falsify the hypothesis finding counter-examples with which it does not hold.

Answer 5

If you want to go for correctness properties of your code, you can, as already mentioned in previous posts, apply some formal verification tools. This is not an easy thing to do, but may still be doable. There are tools like the KeY system capable of proving first-order properties for Java code. KeY has some problems with things like generics, floats and parallelism, but works quite well for most concepts of the Java language. Moreover, you can automatically create test cases with KeY based on the proof tree.

If you are familiar with JML (this is not hard to learn, basically Java with a bit of logic), you could try out this approach. For really critical parts of your systems, verification might really be something to think about; for other parts of the code, testing some of the possible traces with unit testing might already be sufficient, for example to avoid regression problems.

Answer 6

There are tool for formally specifying how your code operates and even tools to proof that they work in that way, but they are far away from unit testing area.

Two examples from the Java world are JML and ESC/Java2

NASA has a whole department dedicated to formal methods.