Skip to main content

Code Coverage in C#.NET Unit Tests - Setting up OpenCover

The purpose of this post is to be a brain-dump for how we set up and used OpenCover and ReportGenerator command-line tools for code coverage analysis and reporting in our projects. The documentation made some assumptions that took some digging to fully understand, so to save my (and maybe others') time and effort in the future, here are my notes.

Our project, which I will call CEP for short, includes a handful of sub-projects within the same solution. They are a mix of Web APIs, ASP MVC applications and Class libraries.

For Unit Tests, we chose to write them using the MSTest framework, along with the Moq mocking framework.

As the various sub-projects evolved, we needed to know more about the coverage of our automated tests. What classes, methods and instructions had tests exercising them, and what ones did not?

Code Coverage tools are conveniently built-in for Visual Studio 2017 Enterprise Edition, but not for our Professional Edition installations. Much less for any Community edition.



In order to calculate code coverage, we learned that we could use third-party tools, downloaded through the NuGet Package Manager. Tools like:

  • OpenCover - a command-line tool for calculating the lines and branches covered by the suite of unit tests;
  • ReportGenerator - a command-line tool for converting the xml output of OpenCover into graphical reports and line-by-line coverage breakdowns.

To install the tools for evaluation and experimentation, without risking contaminating our projects, I created a new, clean Solution called TestCoverage. In Visual Studio 2017, this is:

  • File menu -> New Project...
  • Enter "TestCoverage" as the Name and select an appropriate location
  • Select "Create New Solution" from the Solution dropdown
  • Click OK

Then in this new solution, I used the NuGet Package Manager (right-click the Solution in the Solution Explorer) to find and install the latest OpenCover and ReportGenerator packages. This installs them under the "packages" folder of the TestCoverage solution. I am not sure that is the ideal long-term place, but will do for our purposes of experimentation.

Since these are command-line tools, we built a batch-file that

  1. runs the tests, 
  2. provides the test run results to OpenCover, which writes an XML output file, and 
  3. then runs ReportGenerator against that file.

First, I needed to find where the MSTest.exe file was. Normally, I have been launching it via Visual Studio, but for my command-line batch file to work, I need to launch it from the command-line.

To find it, launch the Visual Studio Developer Console. Press the Windows key, type Developer Command Prompt and select it from the list of options.

From the prompt, type:
> where MSTest.exe

This showed me that, on my workstation, the tool is located here:
C:\Program Files (x86)\Microsoft Visual Studio\2017\Professional\Common7\IDE\MSTest.exe

Close the Developer console and open a regular Command prompt (press the Windows key again, type "cmd" and select the Command Prompt). I did not select to run as Administrator, it was not necessary.

MSTest.exe knows what unit tests to run based on its "/testcontainer" input parameter. Set it to point to the DLL of the unit test project.

Subsets of the tests may be specified using other input parameters. We wanted the entire suite to run so we did not use these options.

A simple command-line MSTest run, output to the console, might look like this:
C:\> \Your\Path\Here\MSTest.exe /testcontainer:\Your\DLL\Path\bin\Debug\Tests.dll

The OpenCover tool uses the OpenCover.Console.exe executable.

  • Set its "-target" parameter to the MSTest.exe.
  • Set its "-targetargs" parameter to pass into MSTest.exe the "testcontainer" parameter used above.
  • Set its "-output" parameter to define the path and / or file name where the OpenCover result xml will be written
  • Set its "-register" value according to your needs. This optional parameter may not be needed at all in your context.

A sample command-line OpenCover run that uses MSTest.exe, runs the tests, calculates the coverage, and puts the results in File.xml might look like this:
C:\> \Your\OpenCover\Path\OpenCover.Console.exe -target:"\Your\Path\Here\MSTest.exe" -targetargs:"/testcontainer:\Your\DLL\Path\bin\Debug\TestProject.dll" -output:\Your\Output\File.xml -register:user

Next, turn the XML into more human-readable and meaningful charts with the ReportGenerator, run again by the comman-line interface.

  • Set its "-reports" parameter to point to the XML file output from the OpenCover tool.
  • Set its "-targetdir" parameter to point to the output folder for the HTML pages, images and other files that will be produced.
  • Set its "-assemblyfilters" parameter to tailor the sets of classes that the report will present. Note that for our purposes, it was preferable to set the filter of tests and reports at this level, rather than earlier in the process.

Sample command-line to run the ReportGenerator:
C:\> \Your\ReportGenerator\Path\ReportGenerator.exe -reports:\Your\Output\File.xml -targetdir:\Your\Report\Output\Folder -assemblyfilters:+*;-TestProject

A minor complaint: ReportGenerator did not indicate when a parameter was not recognized, which led to several wasted minutes trying to debug why the command-line value seemed to be ignored. It turns out I had "-assemblyfilter" not "-assemblyfilters" but the tool gave no hint of the problem.

Finally, for reporting purposes, I wanted to include all of the CEP* sub-projects in the evaluation of coverage metrics, except for the CEP Test project. Since the tests would have 100% coverage, including them would skew upward the metrics for the overall solution. So I created Includes and Excludes settings.

Then all of the command line statements were pulled together into a convenient .BAT file.

Final batch file (edited):
SET coverTool="\MyPath\tools\TestCoverage\OpenCover.4.6.519\tools\OpenCover.Console.exe"
SET testTool="\Program Files (x86)\Microsoft Visual Studio\2017\Professional\Common7\IDE\MSTest.exe"
SET reportTool="\MyPath\tools\TestCoverage\ReportGenerator.3.1.2\tools\ReportGenerator.exe"

SET coverXML="\Coverage\OpenCoverResults\CEPCoverage.xml"
SET coverReportLocation="\Coverage\CoverageReport"

SET testAssembly="\MyPath\CEPSolution\CEPSolution.Tests\bin\debug\CEPSolution.Tests.dll"
SET Includes="+CEP*"
SET Excludes="-CEPSolution.Tests;-CEPExperiments*"

%coverTool% -register:user -target:%testTool% -targetargs:"/testcontainer:"%testAssembly% -output:%coverXML%

%reportTool% -reports:%coverXML% -targetdir:%coverReportLocation% -assemblyfilters:%Includes%;%Excludes%

Popular posts from this blog

Git Reset in Eclipse

Using Git and the Eclipse IDE, you have a series of commits in your branch history, but need to back up to an earlier version. The Git Reset feature is a powerful tool with just a whiff of danger, and is accessible with just a couple clicks in Eclipse. In Eclipse, switch to the History view. In my example it shows a series of 3 changes, 3 separate committed versions of the Person file. After commit 6d5ef3e, the HEAD (shown), Index, and Working Directory all have the same version, Person 3.0.

Scala Collections: A Group of groupBy() Examples

Scala provides a rich Collections API. Let's look at the useful groupBy() function. What does groupBy() do? It takes a collection, assesses each item in that collection against a discriminator function, and returns a Map data structure. Each key in the returned map is a distinct result of the discriminator function, and the key's corresponding value is another collection which contains all elements of the original one that evaluate the same way against the discriminator function. So, for example, here is a collection of Strings: val sports = Seq ("baseball", "ice hockey", "football", "basketball", "110m hurdles", "field hockey") Running it through the Scala interpreter produces this output showing our value's definition: sports: Seq[String] = List(baseball, ice hockey, football, basketball, 110m hurdles, field hockey) We can group those sports names by, say, their first letter. To do so, we need a disc

Java 8: Rewrite For-loops using Stream API

Java 8 Tip: Anytime you write a Java For-loop, ask yourself if you can rewrite it with the Streams API. Now that I have moved to Java 8 in my work and home development, whenever I want to use a For-loop, I write it and then see if I can rewrite it using the Stream API. For example: I have an object called myThing, some Collection-like data structure which contains an arbitrary number of Fields. Something has happened, and I want to set all of the fields to some common state, in my case "Hidden"

How to do Git Rebase in Eclipse

This is an abbreviated version of a fuller post about Git Rebase in Eclipse. See the longer one here : One side-effect of merging Git branches is that it leaves a Merge commit. This can create a history view something like: The clutter of parallel lines shows the life spans of those local branches, and extra commits (nine in the above screen-shot, marked by the green arrows icon). Check out this extreme-case history:  http://agentdero.cachefly.net/unethicalblogger.com/images/branch_madness.jpeg Merge Commits show all the gory details of how the code base evolved. For some teams, that’s what they want or need, all the time. Others may find it unnecessarily long and cluttered. They prefer the history to tell the bigger story, and not dwell on tiny details like every trivial Merge-commit. Git Rebase offers us 2 benefits over Git Merge: First, Rebase allows us to clean up a set of local commits before pushing them to the shared, central repository. For this