xUnit + Unquote testing: report the first item where lists differ enough

Question

I am writing a numerical differential equation solver. In my tests, I need to compare the list of actual values, produced by my code, and a list of expected values computed from the analytical solution, and detect if and when the two lists deviate from each other by a value greater than tolerance.

Here is my code up to now:

module Tests

open System
open Xunit
open Swensen.Unquote

open NumericalIntegrator.Stepping

let isWithinToleranceOf expectedFunc tolerance actual =
    let helper (x, y) =
            expectedFunc x - tolerance <! y
            expectedFunc x + tolerance >! y
            
    actual
    |> List.map helper
    |> ignore

// ... other tests...

[<Fact>]
let ``y' = y`` () =
    let theDiffEq (x, y) = y
    let delta = 5e-4
    let initialConditions = (0.0, 1.0)
    let expectedFunc x = Math.Exp x
    let numSteps = 200u
    let tolerance = 1e-5

    let actualResult = stepMany numSteps theDiffEq delta initialConditions

    actualResult
    |> isWithinToleranceOf expectedFunc tolerance

This code gives me this test failed message:

Un totale di 1 file di test corrisponde al criterio specificato.
[xUnit.net 00:00:00.92]     Tests.y' = y [FAIL]
  X Tests.y' = y [202ms]
  Messaggio di errore:
   

1.039760484 < 1.039760349
false

Expected: True
Actual:   False
  Analisi dello stack:
// [omitted]

L'esecuzione dei test non è riuscita.

I would like to display the following information:

• The first index for which actual is not within tolerance of expected
• The values of expected and actual at this index
• The value of tolerance

I know it should be possible to catch and rethrow the exception generated by xUnit, but I only managed to do so very poorly:

let isWithinToleranceOf expectedFunc tolerance actual =
    let helper (x, y) =
            expectedFunc x - tolerance <! y
            expectedFunc x + tolerance >! y
    
    try
        actual
        |> List.map helper
        |> ignore
    with
    | e ->  printfn "\n\nMy info here\n\n"
            raise e

What should I do?

Thank you for your help.

Answer 1

• use a fucntion which returns bool or Result instead of throwing an exception
• use try/catch within helper, do not surround the entire iteration use mapi instead of map
• to get the current index within iteration
• use failwith to throw own exception or reraise() as an equivalent of throw; in C#; or raise some Xunit exception

where XunitException is handled special in the test explorers and IDEs. You can use it to throw an assertion error, or inherit from it to define own assertions.

    let helper index (x, y) =
      try
        expectedFunc x - tolerance <! y
        expectedFunc x + tolerance >! y
        ()
      with e ->
        // reraise
        // failwithf "exceed tolerance at index %i and values %f and %f" index x y
        XunitException("my message", e) |> raise
 
    actualAndCompareValues
    |> List.iteri helper
    |> ignore

if you explore the Xunit.Sdk namespace, you can discover many exception types you can use here. For instance if you Ctrl+Click on Assert.True() (decompile) it throws Xunit.Sdk.TrueException, which is inherited like this:

TrueException ->
 |_ AssertActualExpectedException
    |_ XunitException
      |_ Exception 

Answer 2

I wouldn't muck around with trying to catch and rethrow the Xunit exception raised by the <! and >! Unquote operators. Instead, you could use List.mapi together with Unquote's test function to capture and raise richer expression messages at the source:

let isWithinToleranceOf expectedFunc tolerance actual =
    let helper index (x, y) =
        test <@ ignore index; expectedFunc x - tolerance < y @>
        test <@ ignore index; expectedFunc x + tolerance > y @>
            
    actual
    |> List.mapi helper
    |> ignore