Tracking Integration Test Coverage with Maven and SonarQube

While the combination of Maven, the Maven Surefire Plug-in, Jenkins, and SonarQube provide fantastic visualization and reporting of unit test coverage out of the box, these tools do not provide a configuration free out of the box solution for collecting and displaying the same metrics for integration test coverage.  Awareness of both integration and unit test coverage provides the development team and product owners with a more accurate picture of test completeness.  This more complete picture can be used to justify lack of testing in certain areas, identify risky portions of the code that are not tested in a production-like environment, and show trends in the quality/completeness of integration tests that were not previously measurable.  This article illustrates how a few simple changes to your Maven configuration can provide this valuable insight.

While SonarQube provides general instructions for configuring SonarQube to ingest integration test code coverage metrics, several aspects of the configuration are not well covered in the article and some aspects of the provided examples require extraneous configuration that is avoidable.  This article provides a more complete example including the use of the Maven Failsafe Plug-in as well as Cargo to demonstrate the runtime-instrumentation of code executing in an external container.  This article also demonstrates how to easily aggregate integration test coverage in SonarQube from multiple integration test modules within your code base.  Parts of this article build on the Maven build configuration described in my earlier article on integration testing Web applications with the Maven Failsafe Plug-in and Cargo.

How It Works

This section provides details on the configuration options and setup used to achieve the goals outlined above.  Skip ahead to the demo section if you simply want to try it out.

JaCoCo Agent Configuration and Multiple Integration Test Modules

JaCoCo is the default code coverage tool used by SonarQube.  The JaCoCo Maven Plug-in provides a convenient means to automatically download the JaCoCo agent JAR and configure a Maven property that will enable runtime instrumentation of the code being integration tested.    The following configuration demonstrates how to automatically download the JaCoCo agent JAR.  The complete details of the JaCoCo Maven Plug-in configuration can be found in the example code parent POM.

      <plugin>
        <groupId>org.jacoco</groupId>
        <artifactId>jacoco-maven-plugin</artifactId>
        <version>0.6.3.201306030806</version>
        <executions>
          <execution>
            <id>prepare-agent</id>
            <goals>
              <goal>prepare-agent</goal>
            </goals>
            <configuration>
              <destFile>${jacoco.out.path}/${jacoco.out.file}</destFile>
              <propertyName>jacoco.agent.arg</propertyName>
            </configuration>
          </execution>
        </executions>
      </plugin>

• Line 9 – This line directs the plug-in to execute the prepare-agent goal.
• Line 12 – This line directs the plug-in to configure the JaCoCo agent to output the coverage file to the desired location.
• Line 13 – This line directs the plug-in to output the the generated configuration arguments to a property called jacoco.agent.arg.  The following snippet represents a possible value of the jacoco.agent.arg property after the plug-in execution completes.

  -javaagent:/home/developerx/.m2/repository/org/jacoco/org.jacoco.agent/0.6.3.201306030806/org.jacoco.agent-0.6.3.201306030806-runtime.jar=destfile=/home/developerx/work/example/sonar-it-example/target/jacoco.exec

Line 12 in the above configuration is of particular importance as it is used to ensure that JaCoCo output from test execution in a multi-module build gets merged into a single file for ingestion by SonarQube.  SonarQube does not support the ingestion of multiple JaCoCo output files in the event that your multi-module build includes more than one module with integration tests.  I often find that 3-tier Web applications often require integration tests at the DAO layer and again at the Web tier.  I prefer to keep the tests closer to the portion of the code that they relate to and that often results in more than one module including integration tests.  Additionally, testing legacy clients against a Web service API often requires multiple modules as only one version of the client classes may be present in the class loader at any given time.  In the event that your application uses a monolithic integration test module, this aspect of the configuration becomes less important.  Ideally SonarQube would work with either approach by ingesting multiple JaCoCo output files in a multi-module build.  Since SonarQube does not support this feature at this time, it is necessary to provide some custom configuration to make things work with multiple integration test modules. The desired configuration must append all output from a multi-module build to a single JaCoCo output file and the file must be deleted before executing a build in order to prevent erroneous reporting of coverage.  The build output folder (target) in the top-level module of your multi-module build gets deleted when executing the clean goal of a Maven build and is accessible from all child modules in the build.  As such, this location represents an ideal location for the merged output file. SonarQube’s examples suggest using a relative path from all modules in a multi-module project to point to the JaCoCo output file location.  Using this approach requires each child module to define a custom configuration for the plug-in with the relative path from that module to the desired location as different child modules may be nested at different levels below the top-level module.  Remembering to redefine the plug-in configuration and configuring the correct relative path is error prone and tedious.  To avoid needing to redefine the plug-in configuration multiple times, the jacoco.out.path property used in the definition of the plug-in’s destFile configuration option.  The complete details of the properties used in the build can be found in the example code parent POM.

<properties>
...
  <jacoco.out.path>${session.executionRootDirectory}/target</jacoco.out.path>
...
</properties>

• Line 3 – session.executionRootDirectory refers to the location in which the build was triggered.  For integration test execution in a multi-module build, a CI server typically executes the build from the top-level module thus providing the desired value for session.executionRootDirectory and allowing us to only configure the JaCoCo Maven Plug-in once in the parent POM.

In the event that your build does not include multiple modules with integration tests, the configuration above may be simplified by leveraging the default value of the destFile configuration option and configuring SonarQube to ingest the default output file.

Configuring SonarQube to Ingest Integration Test Coverage Data

SonarQube provides multiple ways to perform analysis against your code base.  When using a CI server such as Jenkins, the SonarQube Maven Plug-in and the SonarQube Jenkins Plug-in combine to offer a relatively simple mechanism for integrating SonarQube into the continuous integration process. By adding the SonarQube Jenkins Plug-in to your Jenkins installation, triggering a SonarQube analysis of your code is as simple as entering a few configuration options and adding a post build action to your build configuration. A simple Maven configuration change is all that is required to enable the addition of integration test coverage information to the SonarQube analysis. The XML below illustrates this configuration change.

<properties>
...
<sonar.jacoco.itReportPath>${env.WORKSPACE}/target/${jacoco.out.file}</sonar.jacoco.itReportPath>
...
</properties>

• Line 3 – This line defines the property used by the SonarQube Maven Plug-in to locate the JaCoCo output file.

Note that the definition of sonar.jacoco.itReportPath does not leverage the session.executionRootDirectory property as SonarQube does not support the use of the property.  See this Jira ticket for a full explanation of the lack of support for the session.executionRootDirectory property.  Jenkins sets the WORKSPACE environment variable when executing a build.  Referencing this variable in the build allows us to construct an equivalent path to the JaCoCo output file without using the session.executionRootDirectory property.  As developers typically don’t execute the SonarQube Maven Plug-in, the fact that this environment variable is not defined when running a command line build outside of Jenkins does not pose an issue.   Again, if your project does not include multiple modules with integration tests, you can use the default location for the JaCoCo output file as defined by the JaCoCo Maven Plug-in and simply point sonar.jacoco.itReportPath to the default JaCoCo output file.  The complete details of the JaCoCo Maven Plug-in configuration can be found in the example code parent POM.

Collecting Coverage Information for Integration Tests Run In the Failsafe Plug-in

In order for JaCoCo to instrument your code during integration testing, you must configure the JVM running the code under test to use the argument produced by the JaCoCo Maven Plug-in.  The Failsafe Plug-in provides a way to execute integration tests written in JUnit and TestNG during the appropriate lifecycle phases of the Maven build just as the Surefire Plug-in does for unit tests.  For a more complete description of the Failsafe Plug-in, refer to the plug-in documentation or my earlier article about integration testing with the Failsafe Plug-in.  As described earlier in this article, the JaCoCo Maven Plug-in configuration produces a JVM argument that enables the JaCoCo agent to perform runtime instrumentation of your code during testing.  To enable the collection of coverage information for your code as executed by the Maven Failsafe Plug-in, simply include the value of the jacoco.agent.arg property as an argument to the Failsafe Plug-in as shown below.  The complete configuration of the Maven Failsafe Plug-in can be found in the example code parent POM. 

      <plugin>
        <artifactId>maven-failsafe-plugin</artifactId>
        <version>2.16</version>
        <executions>
          <execution>
            <id>verify</id>
            <goals>
              <goal>integration-test</goal>
              <goal>verify</goal>
            </goals>
            <configuration>
              <argLine>${jacoco.agent.arg}</argLine>
            </configuration>
          </execution>
        </executions>
      </plugin>

• Line 12 – This line passes the correct argument to the forked JVM used by the Failsage Plug-in to enable runtime instrumentation of your code.

Collecting Coverage Information for Integration Tests Against Code Running in a Separate Container

Some integration tests exercise your code in the same JVM that executes your integration test code,  just like a unit test.  An integration test for a DAO represents a good example of this type of integration test.  However, some types of integration tests exercise code running in a separate container.  Testing of an HTML based Web application of API testing of a REST Web service represent good example of this type of integration test.  For these types of integration tests, code coverage of the code executing in the remote container is of far more interest than that of the code executing in the Failsafe Plug-in.  In a previous post, I described how to use Cargo in conjunction with the Failsafe Plug-in to perform automated integration testing of a Web application.  This article builds on the setup described in the previous post to provide code coverage metrics for the code executing in Tomcat as controlled by the Cargo Maven Plug-in. The snippet below illustrates the changes to the Cargo plug-in to enable JaCoCo’s runtime instrumentation during integration testing.  The complete details of the Cargo Maven Plug-in configuration can be found in the example code parent POM.

              <configuration>
                <configuration>
                  <properties>
                    <cargo.jvmargs>
                      ${cargo.container.jvmargs.debug}
                      ${jacoco.agent.arg}
                      ${cargo.container.jvmargs}
                    </cargo.jvmargs>
                    <cargo.servlet.port>${cargo.container.port}</cargo.servlet.port>
                    <cargo.tomcat.ajp.port>${cargo.container.tomcat.ajp.port}</cargo.tomcat.ajp.port>
                    <cargo.rmi.port>${cargo.container.rmi.port}</cargo.rmi.port>
                  </properties>
                </configuration>
              </configuration>

• Line 6 – This line passes the correct argument to the forked JVM launched by Cargo to enable runtime instrumentation of your code driven by the integration test.

Putting It All Together: A Running Example

Prerequisites

• To Build In Jenkins with SonarQube
  • SonarQube 3.7+
  • Jenkins 1.5+
  • JDK 1.6+
• To Build From the Command Line
  • Example Source Code
    • Clone from GitHub: git@github.com:DavidValeri/blog-sonar-it.git
  • JDK 1.6+
  • Maven 3.0.3+

Running the Build From the Command Line

Follow the steps below to run the build, including the integration tests, as if you were a developer on the project.  This demonstration illustrates how the collection of integration test coverage metrics does not impact the typical developer’s processes.

• Navigate to the clone of the example Git repository
• Execute the following

  mvn clean install

Running the Build in Jenkins with SonarQube

Follow the steps below to setup Jenkins and SonarQube to build the example application and collect integration test coverage data.  If you already have a SonarQube or Jenkins installation, you may perform the configuration steps on your existing installation; however, these steps describe using the out-of-the-box stand-alone instances of SonarQube and Jenkins.  You may need to modify some steps or configure additional fields if you are running Jenkins and/or SonarQube with an external database or in a container such as Tomcat.

Configuration and Build Execution

• Install and Configure SonarQube.
  • Download SonarQube 3.7 from SonarQube’s download page
  • Extract the SonarQube archive to the location of your choosing
  • Navigate to the bin folder within the extracted SonarQube folders and locate the scripts for your OS
  • Start SonarQube by executing the start script for your OS
  • Navigate to http://localhost:9000
  • Authenticate to SonarQube as the admin user with the username and password admin and admin, respectively
  • Add a user for the Jenkins CI server
    • Click Settings in the upper right corner of the application window
    • Click Users in the blue box on the left side of the application window
    • Complete the “Add new user” form in the yellow box on the right and click the Create button.

• Install and Configure Jenkins
  • Download Jenkins 1.5+ from the Jenkins site
  • Extract/Install Jenkins
    • Use the Jenkins instructions for your platform to install and start Jenkins
  • Navigate to http://localhost:8080
  • Install the SonarQube Jenkins Plug-in following the instructions provided in the SonarQube documentation and restarting Jenkins after the installation.  Note that the SonarQube Jenkins Plug-in may appear as the “Sonar” plug-in in the list of available plug-ins within Jenkins.
  • Configure the SonarQube Jenkins Plug-in and the Maven installation
    • Click Manage Jenkins from the links on the left of the application window
    • Click Configure System from the available configuration options
    • Scorll to the Maven section
    • Click the Maven installations… button
    • Click the Add Maven button
    • Enter “Maven 3.0.4” in the Name field
    • Check the Install automatically button
    • Select 3.0.4 from the Install from Apache menu
    • Scroll to the Sonar section
    • Click the Add Sonar button
    • Enter “Local” in the Name field
    • Click the Advanced… button
    • Enter the username and password of the Jenkins user created earlier in the Sonar account login and Sonar account password fields, respectively
    • Click the Save button
  • Install the Jenkins GitHub Plug-in
    • Using the same process used earlier to install the SonarQube Jenkins Plug-in, locate the GitHub Plug-in from the list of available plug-ins in Jenkins.  Restart Jenkins after installing the plug-in.
  • Create a new Maven based project
    • Click New Job from the links on the left of the application window
    • Enter “Test” in the Name field
    • Select the radio button for Build a maven 2/3 project 
    • Click the OK button
    • Scroll to the Source Code Management section
    • Select the radio button for the Git option
    • Enter “git@github.com:DavidValeri/blog-sonar-it.git” in the Repository URL field
    • Scroll to the Post-build Actions section
    • Click the Add post-build action button and select Sonar from the menu.  The default settings are sufficient for this demo
    • Click the Save button

• Build the project in Jenkins
  • Navigate to http://localhost:8080
  • Click on Test in the project list
  • Click Build Now on the left side of the application window to trigger the build
  • Observe the build results for success.  In the case of a build failure, diagnose the issue before proceeding further.

Examining the Test Coverage Data in SonarQube

Follow the steps described in this section to explore the integration test coverage data available in SonarQube for the example project.

• Navigate to http://localhost:9000
• Click the David Valeri’s Blog : Examples :: Sonar IT project from the Projects widget on the main screen.
• Add the Integration Test Coverage widget
  • Click Configure Widgets in the upper right hand corner of the dashboard
  • Click the Tests tab in the yellow box
  • Click the Add widget button for the Integration Tests Coverage widget
  • Drag the widget to the desired location on the dashboard
  • Click Back to dashboard in the upper right hand corner of the dashboard configuration screen
• Drill down to view integration test coverage details
  • Click on the line coverage percentage value in the Integration Tests Coverage widget
  • Click on one of the classes listed in the right hand column
  • Use the checkbox and menu to view different line and branch level coverage details for the selected class.
• Explore other integration test related metrics
  • Click Hotspots in the blue box on the left side of the dashboard
  • Click Configure Widgets in the upper right corner of the dashboard
  • Click the Hotspots tab in the yellow box
  • Click the Add widget button for the Metric Hotspot widget
  • Drag the widget to the desired location on the dashboard
  • Click the Edit link in the Metric Hotspot widget
  • Explore the unit and integration test metrics available and choose the desired metric from the menu.
  • Click Back to dashboard in the upper right hand corner of the dashboard configuration screen and observe the newly configured widget