Maven: A Developer's Notebook

Reporting Testing Status

Good tests are a major step toward quality software. How do you make this information visible? Two solutions come to mind: sharing a test execution report and showing how much of your code is covered by your tests.

How do I do that?

Let's use the qotd/core subproject as a guinea pig for adding a test execution report. Let's add a failing test in addition to the existing one (testGenerate) to see what happens in that case. Add the following test to the qotd/core/src/test/mdn/qotd/core/QuoteGeneratorTest.java class:

public void testBadTestToShowReportingInCaseOfFailure()
{
    assertEquals("Hello Quote!", new QuoteGenerator().generate());        
}

Add the test report to the qotd/core/project.xml reports section:

<reports>
  <report>maven-junit-report-plugin</report>
</reports>

Let's generate the test report by executing maven site. The generated report is in qotd/core/target/docs/junit-report.html, shown in Figure 4-3.

Figure 4-3. JUnit test report showing number of tests and test results

That's good... but it's not enough! This report will show to your users how many tests you have, and their success rate. However, it does not say anything about how good your tests are and whether they properly test your application. For that you need to run a test coverage tool. Fortunately, several are available (including Clover, JCoverage, and Emma), and they all have Maven plug-ins.^[1]

Let's use Clover by simply adding the Clover report to the reports section in qotd/core/project.xml:

  <reports>
    <report>maven-clover-plugin</report>
    [...]

Run maven site and check the generated Clover reports in qotd/core/target/docs/clover/index.html. Figure 4-4 shows the summary of what your tests cover of the main code.

Figure 4-4. Total test coverage report showing that 88.9% of the code has been tested

Figure 4-5 shows the details of what has been tested and what has not been tested in your main code.

Figure 4-5. Details of what has and hasn't been tested

You can see in the left gutter how many times the code has been executed, and you can see the code that has not been tested. Here it means you have never tested the case when the Really Simple Syndication (RSS) feed cannot be retrieved (when you're offline, for example). This tells you where you should focus your testing effort in the future...

What about...

...the real quality of the tests?

Indeed, test coverage measures do not fully represent the quality of the tests. They just say what part of the code has been covered by the tests. It can happen that a test is wrong, or tests nothing. There is a nice tool called Jester (http://jester.sourceforge.net) which performs incremental mutations on the source code and verifies if the test catches the introduced differences. Then it reports what the tests are really testing. The idea is great, but the tool doesn't have a Maven plug-in yet. You know what you have to do now! That would be a good exercise once you finish reading this book...

---

^[1] We have chosen to cover the Clover plug-in because it has the most features and probably the best Maven integration. You should know that Clover is commercial software (although it's free for open source projects), JCoverage has a GPL version, and Emma is open source and free under a CPL license.