Is there an easier way to handle unit testing a method with too many conditions?

Question

I have a method which has a lot of conditions in it:

public bool IsLegalSomething(Order order)
{
    var item0 = order.Items.SingleOrDefault(x => x.ItemCode == "ItemCode0");
    var item1 = order.Items.SingleOrDefault(x => x.ItemCode == "ItemCode1");
    ...
    var itemN = order.Items.SingleOrDefault(x => x.ItemCode == "ItemCodeN");

    return ((item0.Status == Status.Open) && (item1.Status == Status.Closed)
         && ...
         && (itemN.Status == Status.Canceled));
}

I want to unit test this function, but there are so many conditions that the number of unit tests is crazy if you consider every combination. There are 16 conditions in that return statement, and since each condition is true/false that is 2^16 different combinations I would need to check on. Do I really need to create 2^16 different unit tests here to ensure that every condition is utilized? Mind you, this is a simple example. Some of my functions have complex conditions due to legal requirements:

return (condition0 && condition1 && (condition2 || condition3)
     && (condition4 || (condition5 && condition6)) ...)

By the math of some of my functions, the number of different combinations that the conditions can produce is millions! I looked into Data-Driven Unit Tests (DDUT), as well as Parameterized Unit Tests (PUT), but that just makes it so that the unit test is a "fill in the blanks" style. I still have to supply all of the various combinations and the expected result! For example:

// Parameterized Unit Test
[TestCase(..., Result = true)]  // Combination 0
[TestCase(..., Result = true)]  // Combination 1
[TestCase(..., Result = false)] // Combination 2
public bool GivenInput_IsLegalSomething_ReturnsValidResult(...) { }

If I use MSTest to pull in a datasource (csv for example), I still have the same problem. I have way too many combinations that give different results. Is there an alternative I just am unaware of?

The design may need to be reviewed and refactored. The complexity/difficulty in creating unit tests are an indicator of how clean is the code being tested. The method being tested is doing too much. — Nkosi, Jan 25 '17 at 18:04
@Nkosi, Understandable, but I'm not really sure how to "simplify" legal requirements that have that many combinations. At the end of the day, some class will contain logic which says "16 conditions drive a Boolean value". — michael, Jan 25 '17 at 18:06
You could write a single test method that cycles through generated combinations. Otherwise, I'd consider focusing on the business use cases. In that case, you might want to refactor "IsLegalSomething" into multiple methods each testing a single thing based on those business use cases. — Peter Ritchie, Jan 25 '17 at 18:09
@PeterRitchie: That's exactly the problem, IsLegalSomething is a unit of code which takes 16 conditions and returns 1 result. How can I refactor code that says 16 inputs results in 1 output? — michael, Jan 25 '17 at 18:17
There isn't the concept that each condition being tested is a different legality? What you're effectively trying to do is validate a certain range of 16-bit values result in true and some result in false. Is there a way to order/group those such that true values are contiguous range of 16-bit values? Maybe that would make it easier to test — Peter Ritchie, Jan 25 '17 at 18:22

score 2 · Accepted Answer · edited May 23 '17 at 12:01

While I agree with the comments made about refactoring your code, I think a more concise answer is warranted to explain what exactly is meant by "The design may need to be reviewed and refactored."

Let's look at the following statement: A && B && (C || D);. Normally, you would say that you have 4 inputs @ 2 options/each, or 16 combinations. However, if you refactor things you can reduce down the complexity. This will vary depending on your business domain, so I'm going to utilize a shopping site domain as an example (we don't actually know your business domain).

A: Is New Order
B: Are All Items In Stock
C: Is Customer Elite Member
D: Is Discount Code FREESHIP Used

The reason I chose this scenario is to demonstrate that it is possible that C/D actually references whether or not shipping should be included free of charge.

E: Is Shipping Free

Now, instead of A && B && (C || D) we have A && B && E, which is 3 conditions @ 2 options/each, or 8 combinations. Of course, the composition of E should be tested as well, but C || D only has 2 options @ 2 options/each, or 4 combinations. We have reduced down the number of total combinations from 16 to 12. While that may not seem like much, the scale of this scenario was far less. If you are able to group logical conditions together and reduce things further, you could go from millions of combinations to just a few hundred, which is far more easy to maintain.

Additionally, as a bonus, sometimes your domain logic changes in some aspects, but not in others. Imagine that you decide that Commercial clients also receive free shipping one day, instead of adding another condition to an extremely complex conditional statement, essentially doubling the number of unit tests, you would add that condition to the Is Free Shipping, which would increase the number of combinations for that smaller unit from 4 to 8, but it's better than having a function that has 16 conditions (i.e. 65536 combinations) to 17 conditions (i.e. 131072 combinations).

At this point, unless we know and understand your exact domain, we can only make broad suggestions of redesigning your classes and methods into smaller parts. Also, while mart makes a good point using string length > 5 not needing to test for every string greater than 5 length, I do think that once you have an actual condition reduced down into true/false that combined conditions need to be tested. For example: (String Length > 5) && B && C && D && E has 5 conditions or 32 combinations. You should test all 32. What you shouldn't do is come up with 100 different ways to show that String Length > 5 is true. The reason I say to test all 32 combinations is because while the conditions can be refactored and tested, you still want to test that you are using A && B && (C || D) so you can be sure no one made a typo of A && B || (C && D). Or, in other words, proving the individual conditions does not guarantee that the combination of those conditions was correctly coded.

score 0 · Answer 2 · answered Jan 25 '17 at 18:29

You should refactor the code to test it correctly, each block of conditions should be a function by itself, and that function should be tested individually, then your main function only will need test to check the integration.

Unit testing is not meant to cover all the options, in fact if you are testing true && true, false && false, and so on, you are practically testing && operator and not your logic, you should cover what your function as "unit" does and not all the possible combinations of inputs, just think in a function that check if the length of a string is grater than 5, are you gonna create all the possible strings in the world? or are you just gonna test one that is equal to 5, one with more than 5 characters and one that is lower, maybe you should test null too, but not more than that.

Anyway just to give you a real alternative to what you are asking if you are able to add NUnit to your project you could use https://github.com/nunit/docs/wiki/Combinatorial-Attribute in order to generate all those test that you want.

Is there an easier way to handle unit testing a method with too many conditions?

2 Answers2