66

How do code coverage tools like NCover know what parts of the code were executed and what parts were not?

Hannoun Yassir
  • 20,583
  • 23
  • 77
  • 112

5 Answers5

24

Here's a technical paper on How to Implement test coverage tools for arbitrary languages.

My company builds a family of test coverage tools for Java, C#, C++, PHP, COBOL, PLSQL, ... based on this principle.

Ira Baxter
  • 93,541
  • 22
  • 172
  • 341
9

Quote straight from the NCover FAQ: NCover reports the percentage of branches in the code that have been taken throughout the course of your automated testing. It achieves this by instrumenting the source code at each branch, and writing the 'hit' points to a file. These 'hit' points are then compared to the total possible points that could have been 'hit'.

Hans Olsson
  • 54,199
  • 15
  • 94
  • 116
  • 8
    @Yassir: You might want to clarify you question about that. However, since NCover is open source, you can already see how it does it... I'd suggest seeing if you can look at an as early version as possible, where there will probably be less code and so should show you the structure easier. – Hans Olsson Jun 16 '10 at 08:14
  • 1
    I've actually tried that but the code isn't really clear since the are no comments in the code :) – Hannoun Yassir Jun 16 '10 at 09:23
  • 1
    @Yassir: How unusal for an Open Source project :). There's the PartCover project, maybe that's better http://sourceforge.net/projects/partcover/ – Hans Olsson Jun 16 '10 at 09:31
  • This answer is more like a restatement of the question. It is more saying what code coverage is than how it is possible or how it is accomplished. – still_dreaming_1 Oct 20 '16 at 21:52
7

From this source:

NCover uses the .NET Framework profiler API to monitor an application's execution. When a method is loaded by the CLR, NCover retrieves the IL and replaces it with instrumented IL code

So in short, it hooks itself into the just-in-time compilation.

Not all tools work the same way though. Other tools work by modifying the bytecode of your application after the code has been compiled.

Pete
  • 12,206
  • 8
  • 54
  • 70
6

I know this is question is old but if you are still interested you can see an example of how such instrumentation is performed for .NET applications by looking at the open source project OpenCover.

OpenCover inserts instrumentation points at significant points in the code.

  1. For code line coverage it uses the sequence points taken from a PDB file
  2. For branch coverage it instruments COND_BRANCH instructions by instrumenting the jump target(s) and the next instruction after the branch instruction i.e. no jump.
  3. For method instrumentation it instruments the first instruction of any method.

All of these rules are applied in CoverageInstrumentation.cpp after the appropriate points have been located using Mono.Cecil and passed to the profiler from the console host.

The source code to PartCover is also available (as indicated) but this is much harder to follow but it also uses sequence points from PDBs to determine where it instruments the code.

Shaun Wilde
  • 8,228
  • 4
  • 36
  • 56
5

It requires that you run your tests once with code coverage analysis enables, and then simply counts the number of blocks (that is, scope blocks) covered and compares to the total number of blocks in the project(s) you are testing.

The basic reasoning is that if each possible combination of code blocks is covered, all code paths are covered1. The main argument against putting too much weight in code coverage numbers is that "easy" blocks like getters and setters, that give no real value (and could hardly go wrong...) count just as much as more error-prone blocks of code.

1) As noted by Ira Baxter in a comment, the previous wording of this sentence was incorrect. Please read the comments for some discussion on this.

Community
  • 1
  • 1
Tomas Aschan
  • 58,548
  • 56
  • 243
  • 402
  • 4
    "If each block is covered, all code paths are coverered". False. Imagine four blocks, two pairs A1 A2 B1 B2 each controlled by an if-thenelse A and B. Under the assumption that the if conditions are independent, there are 4 possible paths. With A true, you can exercise B true and false; that will cover A1 B1 B2. With A false and B false, A2 and B2 are covered. With the set of test described so far, A1 A2 B1 B2 (all blocks) are all covered. But the path A2 B1 hasn't been executed. Not all paths are covered. Block/branch coverage != path coverage. – Ira Baxter Jul 06 '11 at 17:33
  • 1
    @IraBaxter I don't think the point is to cover every combination of code execution, but that all blocks of code are covered at least once through. – Brenden Nov 25 '13 at 22:42
  • 3
    @Brenden: usually somebody other than you defines the criteria of "good enough". Some will be satisified with "each block executes at least once"; that's essentially "branch coverage". Some insist that each causal subcondition of any boolean expression is covered; this is the much the reason that airplane software is as good at it is (DO-178B). Others may insist that all paths are covered. The point is there are many criterias, and different audiences for each. They tell you what's acceptable, not the other way around. – Ira Baxter Nov 25 '13 at 22:49
  • @ThomasLycken So do code coverage tools measure if each "possible combination of code blocks" is executed or just code blocks? – Honinbo Shusaku Nov 10 '16 at 13:55
  • 1
    @Abdul: It depends on the tool, and its configuration. I think most tools just measure if a code block was ever executed (but doesn't care which way it got there), but I also think (although I have no reference for it at the moment) that there are tools that can actually detect if each possible path through a function is taken. The fact that most tools *don't* do this (e.g. they only measure whether a code block was ever run or not) is one of the main arguments against using code coverage as an important metric for the quality of your test suite. – Tomas Aschan Nov 12 '16 at 11:28