Coverage Metrics

As you probably know, not all code coverage metrics are the same. In fact, you will probably get slightly different numbers from every tool that you use. The reason for this discrepancy is that there are many variations for breaking code down into sections. The way that this information is reported, however, can affect the numbers you see in significant ways, especially when measuring complicated code. Let's look at some examples.

Line Coverage

Many coverage tools report line coverage, which is probably the most basic coverage metric. Line coverage simply measures whether a particular line of code was executed or not. Below is an example of the results you might get when running a moderately complex bit of code.

    // Line Coverage
{
1     int i = 10;
{
3       DoSomething(); DoSomethingElse();
      }
4     return (i == GetMagicNumber()) ? ItWas10() : IsWasnt();     
    }
    // 4 covered lines / 4 coverage points = 100% coverage

Statement Coverage

Statement coverage is a slightly more specific metric which differentiates when multiple code statements are included on a single line of code. With statement coverage you might see something like this:

    // Statement Coverage
{
1     int i = 10;
{
3 4     DoSomething(); DoSomethingElse();
      }
5     return (i == GetMagicNumber()) ? ItWas10() : IsWasnt();     
    }
    // 5 covered statements / 5 coverage points = 100% coverage

NCover's Sequence-Point Coverage

NCover uses sequence-point coverage as its base coverage number. Sequence-point goes a step further and differentiates between each point where the debugger is able to stop when single-stepping through a code file. NCover uses the compiler's debug symbol database to provide this information, so it is guaranteed to provide the same points that the Visual Studio debugger will use when debugging.

With sequence point coverage, you should see the following, assuming GetMagicNumber() returns 10:

        // Statement Coverage
{
1         int i = 10;
{
3 4         DoSomething(); DoSomethingElse();
          }
8 5 6 7   return (i == GetMagicNumber()) ? ItWas10() : ItWasnt();     
        }
        // 7 covered sequence points / 8 coverage points = 87.5% coverage

As you can see, it's important to understand what information your coverage tool is reporting -- in this case, the first two metrics reported 100% coverage even though not all of the code was being executed. A code coverage metric that is too broad will miss opportunities to highlight uncovered code.

Why do we need the Branch Coverage metric?

Now that you understand sequence-point coverage, let's talk about branch coverage and why it matters. Consider the following two code examples:

{ { throw new EmptyOrderException(); { decimal orderTotal = 0m; { orderTotal += lineItem.SubTotal; } order.Total = orderTotal; order.Save(); } } // 7 covered sequence points / 8 coverage points = 87.5% coverage

We have 87.5% code coverage. Not too bad. Now I'm going to write code that does the same thing, but takes a few extra steps to do it:

{ { throw new EmptyOrderException(); { decimal orderTotal = 0m; { var subTotal = lineItem.SubTotal; var newTotal = subTotal + lineItem.SubTotal; orderTotal = newTotal; } order.Total = orderTotal; order.Save(); } } // 9 covered sequence points / 10 coverage points = 90% coverage

By taking more lines of code to do the same thing, we actually increased our code coverage numbers! If you are paying a lot of attention to the code coverage numbers, then you don't want your code coverage percentages to change depending on how many lines of code you use to write a function. Code coverage percentages should be related to the complexity of the code. This is where branch coverage comes in.

Definition of Branch Coverage

Branch coverage measures the fraction of independent code segments that were executed.

Independent code segments are sections of code that have no branches into or out of them. Put another way, an independent code segment is a section of code that you would expect to execute in its entirety every time it's run.

  // Branch Segment 1
  FunctionA();
  FunctionB();

{
    // Branch Segment 2
    FunctionB();
    FunctionC();
{
    // Branch Segment 3
    FunctionD();
  }

What about exceptions? Asynchronous exceptions such as OutOfMemory or ThreadAbort can happen anywhere and functions that are being called by this segment of code can throw exceptions as well. When determining branch segments, we only consider exceptions thrown from the segment of code itself.

When the segment of code is executed and an exception is thrown within the segment, we consider the segment as uncovered. When the results are combined with sequence point coverage, we can see which portions of the partially executed segment were covered.

  // Branch Segment 1
  FunctionA();
  FunctionB();


{
    // Branch Segment 2
    throw new SomethingNotValidException();
  } 

// Branch Segment 3
  FunctionD();
  throw new NotImplementedException();

// Branch Segment 4
  FunctionE();

Hopefully, this article gives you a good understanding of how branch coverage is calculated and why it's a better overall metric than the line/statement/sequence-point coverage metrics. The branch segmentation approach has a strong theoretical basis, and there are quite a few ways in which it can show you uncovered code that wouldn't be discovered otherwise!