Unit-testing a probability distribution with conditionals

Question

I have a function choose(elems) -> elem that calls rand() which makes it non-deterministic.

To be able to better test this, I figured that I could split this function in two,

generate_choices(elems, ...) -> distribution
choose(distribution) -> elem

where choose() is a thin wrapper around rand() and generate_choices() generates a distribution from which to draw an element. I could then deterministically test that this probability distribution is as expected.

The distribution is uniform but with two conditionals:

1. If there are not enough elems, add a random fallback element uniformly.
2. If there are still not enough elems, add a random default element uniformly.

Some examples:

generate_choices([a, b, c, d], [], []) -> [a, b, c, d]
generate_choices([a, b, c], [fallback1], []) -> [a, b, c, fallback1]
generate_choices([a, b, c], [fb1, fb2], []) -> [a, b, c, (fb1 | fb2)]
generate_choices([a, b], [fb1, fb2], [default1]) -> [a, b, (fb1 | fb2), default1]
generate_choices([a, b], [fb1, fb2], [d1, d2]) -> [a, b, (fb1 | fb2), (d1 | d2) ]
generate_choices([a], [fb1, fb2], [d1, d2]) -> [a, (fb1|fb2), (d1|d2) ]

My question is then: How should I model distribution?

• If I choose a simple list and call rand() from within generate_choices() to fill the fallback and default, then I can only test some deterministic parts of generate_choices().
• If I choose three lists, (elems, fallback, default), then generate_choices() is fully deterministic, but then choice() becomes less trivial and must be tested more thoroughly anyways.

Answer 1

There are at least two ways to test functions which make use of random numbers. You might want to have both kinds of test.

(1) One kind is to set the initial state of the random number generator and generate some examples, and verify by inspection that the examples are correct. Then put those examples into your test script as the expected output, and set the same initial state at the start of the script.

(2) Another kind of test is to generate lots of examples, and verify that on the average, the examples satisfy the properties that are expected. This is a non-deterministic test, since it is possible that for some sequences generated by the random number generator, the test will fail. You'll have to accept some small probability of failure; the good news is that you can make the probability small enough by testing a large number of examples and making the tolerance for the test large enough.

(2a) For example, in the simplest case you give, the input is a sequence and the output is a permutation of that sequence. If you generate a large number of examples, you should find that all permutations have equal frequency, within some tolerance. Obviously this test is limited by the length of the test input, since there are n! permutations of an input of length n.

You can derive a tolerance by considering the distribution of the proportion of each distinct permutation. Each permutation has probability 1/n!, and the expected number of each one is (m times 1/n!) where m is the number of generated permutations. The variance of the number of each permutation is (m times 1/n! times (1 - 1/n!)) and the standard deviation is the square root of that. You can approximate a tolerance interval as (expected number plus or minus a multiple of the standard deviation). You could get a more precise interval by considering the distribution more carefully.

(2b) Another way to test permutations is to look at how many times the first element of the output is equal to the first element of the input, first element of output is equal to second element of input, first element of input is equal to third element of input, ..., second element of input, ..., third element, ... last element of input. This might be feasible for longer sequences than test 2a. Again the game is to figure out the distribution of the expected numbers in each bin and derive a tolerance from that.