Estimate unit tests required in large code base

Question

Our team is responsible for a large codebase containing legal rules.

The codebase works mostly like this:

class SNR_15_UNR extends Rule {
    public double getValue(RuleContext context) {
        double snr_15_ABK = context.getValue(SNR_15_ABK.class);
        double UNR = context.getValue(GLOBAL_UNR.class);
        if(UNR <= 0) // if UNR value would reduce snr, apply the reduction
          return snr_15_ABK + UNR;
        return snr_15_ABK;
    }
}

When context.getValue(Class<? extends Rule>) is called, it just evaluates the specific rule and returns the result. This allows you to create a dependency graph while a rule is evaluating, and also to detect cyclic dependencies.

There are about 500 rule classes like this. We now want to implement tests to verify the correctness of these rules.

Our goal is to implement a testing list as follows:

TEST org.project.rules.SNR_15_UNR
INPUT org.project.rules.SNR_15_ABK = 50
INPUT org.project.rules.UNR = 15
OUTPUT SHOULD BE 50

TEST org.project.rules.SNR_15_UNR
INPUT org.project.rules.SNR_15_ABK = 50
INPUT org.project.rules.UNR = -15
OUTPUT SHOULD BE 35

Question is: how many test scenario's are needed? Is it possible to use static code analysis to detect how many unique code paths exist throughout the code? Does any such tool exist, or do I have to start mucking about with Eclipse JDT?

For clarity: I am not looking for code coverage tools. These tell me which code has been executed and which code was not. I want to estimate the development effort required to implement unit tests.

Should it be `Double` which could be `null` or do you really mean `double`? — Peter Lawrey, Feb 23 '12 at 15:59
For the testing framework, based on your example, this could be useful although I have never used it: http://fitnesse.org/FitNesse.UserGuide.TwoMinuteExample — assylias, Feb 23 '12 at 16:04
@PeterLawrey does not really matter, but I changed it to avoid confusion. — parasietje, Feb 23 '12 at 16:04
The performance difference is not that big in the over scheme of things, but it is clearer IMHO. — Peter Lawrey, Feb 23 '12 at 16:08
Given that the number of test cases may become very large, you might want to consider pairwise testing. There's a very handy tool called PICT that will generate pairwise test cases based on rules you define using a DSL. Here's a thorough introduction to PICT: http://msdn.microsoft.com/en-us/library/cc150619.aspx — Chris B, Feb 27 '12 at 11:51

Ira Baxter · Accepted Answer · 2012-02-27T23:23:00.617

(EDIT 2/25, focused on test-coding effort):

You have 500 sub-classes, and each appears (based on your example with one conditional) to have 2 cases. I'd guess you need 500*2 tests.

If your code is not a regular as you imply, a conventional (branch) code coverage tool might not be the answer you think you want as starting place, but it might actually help you make an estimate. Code T<50 tests across randomly chosen classes, and collect code coverage data P (as a percentage) over whatever part of the code base you think needs testing (particularly your classes). Then you need roughly (1-P)*100*T tests.

If your extended classes are all as regular as you imply, you might consider generating them. If you trust the generation process, you might be able avoid writing the tests.

(ORIGINAL RESPONSE, focused on path coverage tools)

Most code coverage tools are "line" or "branch" coverage tools; they do not count unique paths through the code. At best they count basic blocks.

Path coverage tools do exist; people have built them for research demos, but commercial versions are relatively rare. You can find one at http://testingfaqs.org/t-eval.html#TCATPATH. I don't think this one handles Java.

One of the issues is that the apparent paths through code is generally exponential in the number of decisions since each encountered decision generates a True path and a False path based on the outcome of the conditional (1 decision --> 2 paths, 2 decisions --> 4 paths, ...). Worse loops are in effect a decision repeated as many times as the loop iterates; a loop that repeats a 100 times in effect has 2**100 paths. To control this problem, the more interesting path coverage tools try to determine the feasibility of a path: if the symbolically combined predicates from the conditionals in a prefix of that path are effectively false, the path is infeasible and can be ignored, since it can't really occur. Another standard trick is treat loops as 0, 1, and N iterations to reduce the number of apparent paths. Managing the number of paths requires rather a lot of machinery, considerably above what most branch-coverage test tools need, which helps explain why real path coverage tools are rare.

score -1 · Answer 2 · answered Feb 23 '12 at 15:58

-1

how many test scenario's are needed?

Many. 500 might be a good start.

Is it possible to use static code analysis to detect how many unique code paths exist throughout the code?

Yes. It called a code coverage tool. Here are some free ones. http://www.java-sources.com/open-source/code-coverage

answered Feb 23 '12 at 15:58

Peter Lawrey

525,659
79
751
1,130

1

Code coverage tools that are not specifically path coverage tools do not count unique paths through the code. They count basic blocks, at best. – Ira Baxter Feb 26 '12 at 00:41
Counting unique paths is extremely expensive and difficult to visualise. I guess I didn't take this literally because I couldn't imagine this to be useful. (Given I don't know of any such tool, I might not be alone ;) – Peter Lawrey Feb 26 '12 at 07:52
Now you do: http://testingfaqs.org/t-eval.html#TCATPATH (I have nothing to do with this tool, and have not used it. The point is they exist as commercial tools). – Ira Baxter Feb 26 '12 at 19:59
@IraBaxter please add this as an answer so I can award you points and even a bounty :-) – parasietje Feb 27 '12 at 10:46

Estimate unit tests required in large code base

2 Answers2