.NET Core Code Quality with Coverlet and SonarQube, Part 2

By Sandeep Anantha

In a previous blog, I introduced SonarQube, a tool that can identify code smells, bugs, and vulnerabilities. SonarQube can increase .NET Core code quality, especially when used with Coverlet. In this blog, I show how developers can put these tools in place to improve quality.

Source Code Scanning using SonarQube (SQ)

Prerequisites

  • .NET Core 2.0
  • XUnit
  • Docker
  • VS Code
  • Coverlet
  • SonarQube
  • All the above tools are very popular and need no introduction except for Coverlet and SonarQube.

    Coverlet is a cross-platform code coverage tool for .NET Core. As per the official documentation, Coverlet generates code coverage information by going through the following process:

  • Locates the unit test assembly and selects all the referenced assemblies that have PDBs.
  • Instruments the selected assemblies by inserting code to record sequence point hits to a temporary file.
  • 2. After Tests Run

  • Restores the original non-instrumented assembly files.
  • Reads the recorded hits information from the temporary file.
  • Generates the coverage results from the hits information and writes it to a file.
  • For more information on Coverlet, please refer to the official documentation.

    You can choose to generate reports in a wide variety of formats such as opencover, lcov, json, and cuberta — opencover is the most popular.

    Install Coverlet either globally across the projects or locally scoped to a project (using package reference) using the following commands:

    dotnet tool install — global coverlet.console. //global

    dotnet add package coverlet.msbuild //local

    To generate coverage reports, you can use Coverlet directly by executing the following command:

    Syntax:

    coverlet ASSEMBLY — target TARGET — targetargs TARGETARGS — format opencover

    Example:

    coverlet /path/to/test-assembly.dll — target “dotnet” — targetargs “test /path/to/test-project” — no-build — format opencover

    SonarQube is a web service that can be installed on a central server so that it is available across the enterprise or department. If SonarQube is not installed and is not available, we can use Docker to set up a local instance of it.

    Sample Project

    For the purpose of this blog, we will create a class called SimpleMath in a class library, and we will also create tests around the SimpleMath class in a separate test project. Add a SimpleMath class with add, subtract, and divide methods to the Math class library, and add a test class to the test library.

    1. Create Parent Folder

    mkdir Mathcd Math

    2. Create Math Project

    mkdir Mathcd Math dotnet new classlib

    Add a simple class that has basic math operations as methods. You can pull down the sample code here.

    author

    Note: There is a condition in the divide method (DividebyZero) that is executed only when the second parameter is 0. If you never call this method passing 0 for the second parameter from the test class, the coverage will never be 100%.

    Note: The multiply method is intentionally commented out to demonstrate the coverage metric.

    3. Create Test Project for Math

    cd ..mkdir Math.Testcd Math.Testdotnet new xunit

    4. Add a Reference to the Above Math Project

    dotnet add reference ../Math/Math.csproj

    5) Add a Test File with the Code Below That Tests the Math Methods Created Above.

    using System;using Math.Api;using Xunit;

    author

    6. Create the Solution File

    cd ..dotnet new slndotnet sln add Math/Math.csprojdotnet sln add Math.Test/Math.Test.csproj

    7. Create Coverage Reports

    Once the code base is ready, and the corresponding test project is created and built, it is time to generate coverage reports off of the test project.

    Running Coverlet

    With setup out of the way, let’s dive into Coverlet. We can integrate coverlet right into msbuild by installing these 2 packages to your test project.

    cd Math.Testdotnet add package coverlet.msbuild — version 2.0.1dotnet add package FluentAssertions — version 5.0.0

    Now, msbuild will generate coverage reports when you pass the right options.

    cd ..dotnet test Math.Test/Math.Test.csproj /p:CollectCoverage=true \/p:CoverletOutputFormat=opencover

    Once the build runs, you should see the below output on the console. In addition to this, a file(‘coverage.opencover.xml’) will also be created in the test folder that has a coverage report. I used the opencover format which is very popular, but you can also use various other formats. Refer to the Coverlet documentation for more information.

    author

    You can clearly see that coverage is 100%. But to test it, if you comment out the DivideTest_ThrowsException case that covers the divide by zero case, and run it again, you will see the coverage drop by about 10%. This is happening as the Dividebyzero branch code is not tested by any method and is not covered, hence Coverlet marks it as not covered.

    author

    You can clearly see that Coverlet is using tests to verify the coverage of the code.

    Integration with SonarQube

    Now that coverage reports have been generated, we have to send these reports to SonarQube for further analysis of code smells and vulnerabilities. For this demo, we will use a local instance of SonarQube. Since installing SonarQube is a tedious process, we will set up a local Docker instance of SonarQube by running the command below:

    docker run -d — name sonarqube -p 9000:9000 -p 9092:9092 sonarqube

    P.S.: If you run into a port conflict, please change the host port number (9000) to something else, but the container port number should remain the same

    We need to install the SonarScanner tool to analyze .NET assemblies.

    dotnet tool install — global dotnet-sonarscanner

    After it is installed, execute the below commands to start SonarScanner and to send a report result to SonarQube:

    a) dotnet test Math.Test/Math.Test.csproj /p:CollectCoverage=true /p:CoverletOutputFormat=opencover

    b) dotnet build-server shutdown

    c) dotnet sonarscanner begin /k:Math /d:sonar.host.url=http://localhost:9000 \/d:sonar.cs.opencover.reportsPaths=Math.Test/coverage.opencover.xml \/d:sonar.coverage.exclusions=”**Test*.cs”

    d) dotnet build

    e) dotnet sonarscanner end

    Here is a short description of the above commands:

  • Since Coverlet is integrated with msbuild, Coverlet switch parameters are passed in the dotnet test command directly, which are then passed down to Coverlet. So the .NET test will piggyback on Coverlet to generate coverage reports. This command will generate coverage.opencover.xml in Math.Test folder.
  • Shutdown all msbuild tasks to not interfere with these project folders.
  • As mentioned, SonarScanner feeds off of coverage reports, so we have to pass path as a parameter. This command starts the scan process and sends results to the local instance of the SonarQube. We also have to exclude the tests project as we do not want to measure coverage on tests itself.
  • Build the solution. This will create a SonarQube folder and will save analysis reports.
  • End the SonarScanner process
  • Voila, that’s it. Browse the SonarQube dashboard (http//localhost:9000/) and you should see the coverage results.

    You can make changes to the code, repeat the process, and see it reflected in the SonarQube dashboard.

    author

    Quality Gates

    An additional feature of SonarQube is its ability to mark a build as pass or fail based on the quality gate and SonarQube comes with default quality gates for popular languages. Quality gates in SonarQube are a set of rules used to enforce quality standards. If the code has not met the defined rule, SonarQube will mark it as a fail. We can also customize these rules per organizational divisions and business unit.

    For details on SonarQube quality gate, click on the quality gates tab (http://localhost:9000/quality_gates/show/1)

    Take a look at the definition of a default quality gate. One of the rules is that “Coverage on New Code” should be greater than 80%. What this means is that if we add new code without adding corresponding tests then the coverage for new code drops below 80%. Hence, it will be treated as an error and the quality gate will fail.

    To induce this error, let’s add a new multiply method (or uncomment if you have the same sample code we started with) to our SimpleMath class without adding any tests to it.

    public decimal multiply (int a, int b) {return a * b;}

    If you repeat the same process again you will see overall coverage drop by 20%. But the important thing to note is that coverage on new code is less than 80%. The SonarQube quality gate fails with a red flag.

    author

    Clicking on the Math Project link, we get into the details. Here you can clearly see that the quality gate fails as the coverage is below 80%.

    author

    If you add tests for the ‘multiply’ method as well, this Quality Gate will pass. This is how SonarQube enforces quality standards.

    In addition to this, SonarQube will also fail in case the reliability/maintainability or security rating falls below A.

    Quality Gates can be integrated into automated CI/CD (Jenkins/Concourse/Cruise control) and deployment can be stopped if it fails. If there is no off-the-shelf resource to query quality gate status, SonarQube also exposes this status over REST.

    P.S.: Source code for this demo can be cloned from this link.

    About the author

    Sandeep Anantha is an Agile software engineer at TribalScale.

    TribalScale is a global innovation firm that helps enterprises adapt and thrive in the digital era. We transform teams and processes, build best-in-class digital products, and create disruptive startups. Learn more about us on our website. Connect with us on Twitter, LinkedIn & Facebook!

    Visit Us on Medium

    You might also be interested in…