How do you run async tests in FsCheck?

Question

How can I get repeatable async tests with FsCheck? Here is a sample code that I run in FSI:

let prop_simple() = gen {
    let! s = Arb.generate<string>
    printfn "simple: s = %A" s
    return 0 < 1
}
let prop_async() =
    async {
        let s = Arb.generate<string> |> Gen.sample 10 1 |> List.head
        // let! x = save_to_db s // for example
        printfn "async: s = %A" s
        return 0 < 1
    } 
    |> Async.RunSynchronously

let check_props() = 
    //FC2.FsCheckModifiers.Register()
    let config = 
        { FsCheck.Config.Default with 
            MaxTest = 5
            Replay = Random.StdGen(952012316,296546221) |> Some
        }
    Check.One(config, prop_simple)
    Check.One(config, prop_async)

The output looks something like this:

simple: s = "VDm2JQs5z"
simple: s = "NVgDf2mQs8zaWELndK"
simple: s = "TWz3Yjl2tHFERyrMTvl0HOqgx"
simple: s = "KRWC92vBdZAHj6qcf"
simple: s = "CTJbQGXzpLBNn0RY6MCvlfUtbQhCUKm9tbXFhLSu0RcYmi"
Ok, passed 5 tests.
async: s = "aOE"
async: s = "y8"
async: s = "y8"
async: s = "q"
async: s = "q"
Ok, passed 5 tests.

Another run can look like this:

simple: s = "VDm2JQs5z"
simple: s = "NVgDf2mQs8zaWELndK"
simple: s = "TWz3Yjl2tHFERyrMTvl0HOqgx"
simple: s = "KRWC92vBdZAHj6qcf"
simple: s = "CTJbQGXzpLBNn0RY6MCvlfUtbQhCUKm9tbXFhLSu0RcYmi"
Ok, passed 5 tests.
async: s = "g"
async: s = "g"
async: s = "g"
async: s = ""
async: s = ""
Ok, passed 5 tests.

So prop_simple() works fine & is repeatable (given StdGen(952012316,296546221)).

But prop_async() is not repeatable & seems to generate the same strings over and over.

Also, is there a better way to write prop_async()?

Answer 1

FsCheck's behavior doesn't really have anything to do with the async here, but rather with the fact that inside the async you're using Gen.sample. Gen.sample picks a new time-based seed for every invocation - so the behavior of it insides an FsCheck property is not reproducible. In other words you shouldn't ever use it inside a property, it's there just for exploratory purposes when you're writing a new generator. Since the seed is time-based, and your property is very small, multiple invocations will use the same seed and so you see the same values. As an example, here is a property without any async that has the same behavior:

let prop_simple2() =         
    let s = Arb.generate<string> |> Gen.sample 10 1 |> List.head
    // let! x = save_to_db s // for example
    printfn "simple2: s = %A" s
    0 < 1

prints e.g.

simple2: s = "nrP?.PFh^y"
simple2: s = "nrP?.PFh^y"
simple2: s = "nrP?.PFh^y"
simple2: s = "nrP?.PFh^y"
simple2: s = "nrP?.PFh^y"
Ok, passed 5 tests.

Now as for how to write an async property, I'd keep the asynchrony inside the property and then resolve it using Async.RunSynchronously to a normal value.As a variant on your example:

let prop_async2 =
    gen {
        let! s = Arb.generate<string>
        // let! x = save_to_db s // for example
        let r = 
            async {
                printfn "async2: s = %A" s
            }
            |> Async.RunSynchronously
        return 0 < 1
    }

Which has a deterministic output. (Note also if you're already creating a Gen<'T> instance you don't need to make the property a function. You can, but that just means that FsCheck will generate 100 values for the unit type (these values are of course all () which is effectively null, so it doesn't hurt but is a small performance improvement.)

You can also do it the other way round:

let prop_async3 =
    async {
        let r = gen {
            let! s = Arb.generate<string>
            printfn "async3: s = %A" s
            return 0 < 1
        }
        return r
    }
    |> Async.RunSynchronously

A few gotchas to be aware of.

• Sequential asynchronous code should generally pose little problems, but read on.

• Asynchronous and concurrent code may run into issues like munn said in the comments, i.e. multiple threads/tasks using the same value. Also reproducibility will be affected. You can maybe carefully write your property code so you don't run into this (e.g. by having a prelude in your properties where all necessary values are first generated in a sequential fashion, then kick off the asynchronous functions), but it needs some work and thought.

• If you override Arbitrary instances using Arb.register they will be overridden in a thread local way; i.e. they won't be propagated to a sequence of async Tasks. My advice is to just don't do that. Registered Arbitrary instances are essentially mutable static state and that generally doesn't play all that nice with concurrency.

Taken together I think async properties are definitely possible but it's definitely somehwat of an uphill battle in v2. FsCheck 3 (currently in alpha) supports async and multithreaded execution directly.