Test coverage for many AND/OR conditions in one statement

Question

Example statement:

if (conditionA && conditionB && conditionC && conditionD) {
    return true;
}

I could write unit tests for all 2^4 combinations, but that would easily go out of hand if more conditions are added.

What should be my unit testing strategy to cover all conditions for a statement like this? Is there any other way to make the code more robust?

Answer 1

The way I see this scenario is 1 happy path, and 4 potential points of failure. If each condition is pivotal to allowing true to be returned, then it would be reasonable to write:

1. A single happy-path unit test, the only case where the logic returns true. And
2. A unit test for each variable that could cause the check to fail, asserting that the single variable has the power to prevent the condition from passing.

I understand it's possible to write logic that passes these checks, but actually returns true when multiple variables are false... but I really wouldn't worry about cases like that unless you're working on a spaceship or something where life / death is involved. In almost all cases, the tester is just testing that the implementation fails on the failure of any variable.

Answer 2

A lot has been written about this topic, and your question seems to call for MC/DC.

There is a predicate consisting of multiple conditions that leads to a decision. Well known coverage criteria applied to the predicate in the question include:

1. Decision coverage: Ensure the overall predicate is once true, and once false.
   This leads to two test cases, for example (T,T,T,T) and (F,T,T,T).

2. Basic condition coverage: Ensure each condition is both true and false.
   This can also be achieved with two test cases: (T,T,T,T) and (F,F,F,F).
   Note that basic condition coverage need not imply decision coverage (example: "P AND Q" with test cases (T,F) and (F,T) meets basic condition coverage, but both evaluate to F, so does not achieve 100% decision coverage).

3. Modified Condition / Decision Coverage (MC/DC). Combination of decision and basic condition coverage, "modified" so that it also requires that each condition must individually determine the outcome. The answer by Edwin Buck is a valid MC/DC cover (TTTT, FTTT,TFTT, TTFT, TTTF).
   In general, with N conditions MC/DC requires N+1 test cases as opposed to 2^N. As such, it strikes a good balance between rigor (each condition tested) and efficiency (testing all of 2^4 may not be necessary). The intuition behind this is exactly the reasoning in the answer by Adam Bates.

4. Full condition coverage: Test all 2^N possible combinations.

Answer 3

You might not need to do all 2^4 conditions since, for example, if A is false, the other conditions aren't even checked. You might be able to get away with just 5

    A   B   C   D
    F   X   X   X
    T   F   X   X
    T   T   F   X
    T   T   T   F
    T   T   T   T

But as another-dave said, depending on your code, you might not need to test all of your conditions. Think about the objective of your test and see what is appropriate

Edit: change suggested by avandeursen

Answer 4

I would recommend the following approach

                       A B C D
testTypicalCall()   -> T T T T
testAisFalseFails() -> F T T T
testBisFalseFails(  -> T F T T
testCisFalseFails() -> T T F T
testDisFalseFails() -> T T T F

This captures the four independent ways you might fail, and one can deduce that if two of these ways were to occur in combination, then at least one of your failure tests should trigger.

It also is robust against rearrangement of A, B, C, and D in future refactoring of if statements, and doesn't rely on short-circuiting logic to ensure that the failing condition is captured. (Justin's answer is good too, but by selecting true values for the unchecked values in his solution, you increase the expressive power of the test and protect against error messages indicating the wrong non-true option, should you decide to somehow report which option is false).

Answer 5

As an Intro to my answer, I would like to explain again, why we do software testing. There is a really big misunderstanding by most test folks.

• We do NOT test software to show that it is error free (That is impossible. Already for minor complex software)
• We perform a “constructive test”, to prove that the functionality is acceptable and requirements or features are “working”.
• And most important: We try to find as much as possible errors (“Destructive test”). We will never find all errors. SRGM can be applied to show, how deep testing should go.

Then, and this is already an answer to a part of your question “What should be my unit testing strategy?”

I will cite Automotive SPICE (PAM 3.1), a well-known and proven process model, Process SWE.4, Software Unit Verification:

“The purpose of the Software Unit Verification Process is to verify software units to provide evidence for compliance of the software units with the software detailed design and with the non-functional software requirements.”

Another set of unit test descriptions, for software with higher demands for quality and especially safety, can be taken from ISO 26262 “Road vehicles — Functional safety —”, Part 6: Product development at the software level”, chapter 9, tables 10, 11, 12

Methods for software unit testing

Requirements-based test
Interface test
Fault injection test
Resource usage test
Back-to-back comparison test between model and code, if applicable

Methods for deriving test cases for software unit testing

Analysis of requirements
Generation and analysis of equivalence classes
Analysis of boundary values
Error guessing

And now the most important, answering the second part of your question, you should do a structural coverage analysis (NOT a structural test), to evaluate the completeness of test cases and to demonstrate that there is no unintended functionality. Do never mix up structural test and structural coverage.

So, you will test and check, if the requirements are covered and you will do a structural coverage measurement to prove that. If the coverage result is too low, then you will add more test cases.

The recommended coverage metric is MCDC.

Of course, you can select also one of the many others Coverage methods. But then you should give a rationale in your test strategy, why you do this.

Looking at your question again:

if (conditionA && conditionB && conditionC && conditionD) 
{
    return true;
}

It seems that you are asking for a recommendation for a structural test. I will also answer this question, but be aware that with this method, you are just testing, if the complier works correctly.

For the boolean expression at hand (and other more complex expressions), you might never find one of the following errors:

Error classes

Expression Negation Fault (ENF)
Sub-Expression Negation Fault (SENF)
Sub-Expression Omission Fault (SEOF)
Literal Negation Fault (LNF)
Literal Omission Fault (LOF)
Literal Reference Fault (LRF)
Literal Insertion Fault (LIF)
Operator Reference Fault (ORF)
Stuck-at-1 Fault (SA1) 
Stuck-at-0 Fault (SA0)
Parenthesis Insertion Fault (PIF)
Parenthesis Omission Fault (POF)
Parenthesis Shift Fault (PSF)

Remember, what I said in the beginning, testing should find errors (destructive test). Otherwise you will maybe miss the above-mentioned errors.

Example:

If your requirements originally intended to have “OR”s, instead of “AND”s in your expression (Error class ORF), using Condition Coverage, with test vector “TTTT” and “FFFF”, will give you 100% condition coverage and 100% decision or branch coverage. But you will not find the bug. MCDC would reveal the problem.

You also mentioned the possibility to test all combinations (MCC, Multiple Condition Coverage). For 4 variables, this is OK. But for more variables the test execution duration will grow geometrically. This is not manageable. And that was one of the reasons, why MCDC has been defined.

Now, lets assume that your example statement is correct and coming back to the definition for test cases for a structural test, based on MCDC, for your expression.

There are several definitions available, mostly talking about “Unique Cause” MCDC, neglecting the fact, that meanwhile also “Masking MCDC” and “Unique cause + Masking MCDC” are certified and approved criterions. For those you need to forget about all the tutorials starting with a BlackBox view on a truth table. Talking about structural coverage or test, it should be clear, that only a WhiteBox view will work. And if you happen to develop in a language with boolean short cut evaluation (like for example in Java, C or C++), it will even be more obvious that a WhiteBox view is mandatory.

---

For your boolean expression/decision (“abcd”), and applying boolean short cut evaluation, there are overall 16 Unique Cause MCDC test pairs:

1   Influencing Condition: 'a'  Pair:  0, 15   Unique Cause
2   Influencing Condition: 'a'  Pair:  1, 15   Unique Cause
3   Influencing Condition: 'a'  Pair:  2, 15   Unique Cause
4   Influencing Condition: 'a'  Pair:  3, 15   Unique Cause
5   Influencing Condition: 'a'  Pair:  4, 15   Unique Cause
6   Influencing Condition: 'a'  Pair:  5, 15   Unique Cause
7   Influencing Condition: 'a'  Pair:  6, 15   Unique Cause
8   Influencing Condition: 'a'  Pair:  7, 15   Unique Cause
9   Influencing Condition: 'b'  Pair:  8, 15   Unique Cause
10  Influencing Condition: 'b'  Pair:  9, 15   Unique Cause
11  Influencing Condition: 'b'  Pair: 10, 15   Unique Cause
12  Influencing Condition: 'b'  Pair: 11, 15   Unique Cause
13  Influencing Condition: 'c'  Pair: 12, 15   Unique Cause
14  Influencing Condition: 'c'  Pair: 13, 15   Unique Cause
15  Influencing Condition: 'd'  Pair: 14, 15   Unique Cause

Resulting in a recommended MCDC test set (There are more than one solutions):

Test Pair for Condition 'a':    0  15   (Unique Cause)
Test Pair for Condition 'b':    8  15   (Unique Cause)
Test Pair for Condition 'c':   12  15   (Unique Cause)
Test Pair for Condition 'd':   14  15   (Unique Cause)

Test vector: Final Result: 0 8 12 14 15

 0:  a=0  b=0  c=0  d=0    (0)
 8:  a=1  b=0  c=0  d=0    (0)
12:  a=1  b=1  c=0  d=0    (0)
14:  a=1  b=1  c=1  d=0    (0)
15:  a=1  b=1  c=1  d=1    (1)

---

Without boolean short cut evaluation, you have, very obviously, only 4 Unique Cause MCDC test pairs:

1  Influencing Condition: 'a'  Pair:  7, 15   Unique Cause
2  Influencing Condition: 'b'  Pair: 11, 15   Unique Cause
3  Influencing Condition: 'c'  Pair: 13, 15   Unique Cause
4  Influencing Condition: 'd'  Pair: 14, 15   Unique Cause

Leading to one deterministic solution:

Test vector: Final Result: 7 11 13 14 15

 7:  a=0  b=1  c=1  d=1    (0)
11:  a=1  b=0  c=1  d=1    (0)
13:  a=1  b=1  c=0  d=1    (0)
14:  a=1  b=1  c=1  d=0    (0)
15:  a=1  b=1  c=1  d=1    (1)

---

I hope, I could shed some more light on the issue.

If you want to explore MCDC in more detail, with tool support, you may have a look into

MCDC