Howto validate correctness of functions which use random?

Question

There are some application domains(e.g. GameDev) in which a lot of functions should be created using random values for producing their output. One of examples is presented below:

def generate_key(monster_key_drop_coef):
    key_letters = string.ascii_uppercase
    rand = random.random()
    if monster_key_drop_coef < rand:
        return None

    button = {}
    button["type"] = random.choice([1,2,3])
    button["letter"] = random.choice(key_letters)
    return button

This function generates the item's drop based on several random operations. The problem appears if you want to validate automatically correctness of this function. Generated values are not deterministic and writing of regression tests seems to be impossible.

My questions are:

Is this possible to write useful regression tests for this type of functions?
Is there any general approach for creating some other type of tests in this case?

There are entire books written on this subject. Example: http://www.johndcook.com/Beautiful_Testing_ch10.pdf — Hans Z, Jun 11 '12 at 16:56
Mocking out the RNG to deliver known results (and then testing with specific outputs for which the desired outcome is known) is one approach. — Charles Duffy, Jun 13 '12 at 18:42

score 3 · Answer 1 · answered Jun 11 '12 at 16:52

One of useful unit-tests is presented below:

def test_generate_key():
    button_list = []
    for _ in range(1, 1000):
        button_list.append(generate_key(0.2))

    is_all_none = True
    is_not_none = False
    for key in button_list:
        is_all_none &= (key is None)
        is_not_none |= (key is not None)

    assert is_all_none == False
    assert is_not_none == True

It validates function signature, cover all lines of function's code(good probability) and will pass in 99.999% cases. Also validated that function produces some drop at least one from 1000 and sometimes doesn't generate drop. 0.2 is probability of an item's drop.

score 2 · Answer 2 · answered Jun 24 '12 at 16:20

I would rewrite the function to use dependency injection (the random number generator is passed as a parameter to the function). Then you can pass a mock of a random number generator to test your function with different deterministic "random" inputs.

Of course you can also test your assertions which don't depend on the results of the call to random. Such as:

The functions returns None or a dict with the keys "type" and "letter".
If a dictionary is returned the values are of the appropriate type and range.

I would never write a unittest that has non-deterministic results, even 1 in a thousand. I care about every test failure, and stochastic results would be unsettling. You would be better off encapsulating your randomness, so that the function can be tested independent of the random number generator.

Howto validate correctness of functions which use random?

2 Answers2