Unit Testing with Pex and Code Contracts

Question

I have the following method:

public static DateTime SubQtrs( this DateTime dt, int qtrs )
{
    Contract.Requires( qtrs > -1 );
    Contract.Requires( ( qtrs * 3 ) >= -120000 && ( qtrs * 3 ) <= 120000 );
              // do something
     }

I created a paramertized unit test and used Pex to come up with some unit test. As expected, Pex tested the contract constraints and passed in a qtrs value that violated the contract. I told the unit test to expect the exception with: [ExpectedException(typeof(TraceAssertionException))] and everything passed.

Now when I just run the unit test with the MS Test Harness (without Pex), I get an error: Descrtipion: Precondition failed: (qtrs * 3 >= -120000 && ......

It gives me the stack trace basically telling me the contracthelper failed. Then I'm given the choice to Abort, Retry, or Ignore.

Either way, the test fails and I get: Failed SubQtrsThrowsTraceAssertionException818 DGALib.Extensions.Tests Test method System.ExtensionMethodsTest.SubQtrsThrowsTraceAssertionException818 threw exception System.ArgumentOutOfRangeException, but exception Microsoft.Pex.Framework.Exceptions.TraceAssertionException was expected. Exception message: System.ArgumentOutOfRangeException: Months value must be between +/-120000. ...

Why is the unit test not recognizing the code contract error anymore?

score 0 · Answer 1 · answered Dec 12 '11 at 21:11

It sounds like you're not running the contract rewriter on the assembly when running the unit test manually, so that the Requires contracts never result in exceptions. It instead sounds like you have some manual checking code inside your method that throws ArgumentOutOfRangeException, which is the correct behavior (at runtime, with no contract rewriting), but doesn't match your expected exception for the test. Make sure that you have the Code Contracts rewriter installed and make sure the Project settings include rewriting with exceptions.

Alternately, use the Requires overload that specifies an Exception type so that it matches the internal check. In this case, you might even remove the other explicit exception, as long as you know you plan to use rewriting.

Contract.Requires<ArgumentOutOfRangeException>( ( qtrs * 3 ) >= -120000 && ( qtrs * 3 ) <= 120000 , "qtrs");

I did check my contract settings first. My test project has the Perform Runtime Contract Checking set to Full. Can there be another reason why the contract is not getting picked up. I've though about have the contract throw a specific exception as described above, but shouldn't the test behavior work the same whether it's run by Pex or the MS Test Unit itself? The ArgumentOutOfRange exception is getting thrown by the DateTime.Add method. — Chris, Dec 13 '11 at 01:09
@Chris, hmm, my suspicion is still that something about running it from the unit test runner is causing it to bypass the contract rewriting step. — Dan Bryant, Dec 13 '11 at 14:53

score 0 · Accepted Answer · answered Jan 04 '12 at 03:14

It seems the tests that were throwing up that screen were missing [HostType("Moles")].

I compared a test that was causing the problem with another test that was also supposed to throw a trace assertion and wasn't causing a problem. The only difference was the one causing the problem was missing [HostType("Moles")]. As soon as I added that attribute, the test worked.

There error must about the contracts must have been bubbling up from somewhere else and sending me on a wild goose chase.

Pex must have created that test before I started to add moles to the rest of the testing scenarios.

Unit Testing with Pex and Code Contracts

2 Answers2