Using the JaCoCo integration
The code coverage feature in GUIdancer and Jubula uses JaCoCo (a Java Code Coverage library available under the Eclipse Public License). Code coverage monitoring can be added to an Application Under Test (AUT) in the AUT Configuration dialog for Swing and RCP AUTs, as described in the user manual.
Viewing the report
In the Reporting Perspective, you can see the code coverage report for each test run:
The philosophical part…
A recurring issue with code coverage, though, is what to use it for. One of the first things that comes to mind about code coverage was this blog entry. Experience shows that it is a good way of explaining that the numbers gathered by code coverage are not the aim of the exercise. The absolute values delivered by a monitoring report can’t be used as a guide to quality, nor should they be formulated as an aim for the team.
It’s not about the absolute numbers
What code coverage numbers or percentages tell us is only what lines, methods, classes or decisions have been executed. It tells us nothing about whether they were executed correctly, or whether the test that caused these things to be executed was a good one. It can be far too easy to attach too much importance to a simple number removed from its context – and this can drive the wrong kind of behavior in our processes.
So what is it good for?
It’s pleasing to have good, high percentages for code coverage, but it’s not the absolute values of the individual percentages that we should be looking at. Instead, there are two other things we should be looking at.
The trend of the lines
The first thing of interest is to move away from individual numbers and look at the trend, most especially the development over time of amount of tests : code coverage : amount of code. The graph generated by GUIdancer and Jubula shows the first two numbers as lines over time. The report below shows how code coverage can increase with the addition of tests.
Using graphs such as this, we can see whether a new test added increases the coverage or not. If so, then if we assume no large additions or subtractions from the amount of code, we can reason that the test has executed other lines than executed by previous tests.
If the code coverage value does not increase, it could mean that new code has been added, thus reducing the overall percent of the code coverage. Looking at the detailed report would tell us whether new code was executed or not (more on this in the next section). Even if the amount of code had remained the same, and therefore new lines of code weren't actually executed, that doesn't necessarily mean that the new test isn’t performing an important use case or workflow through the application, just that the lines executed have remained the same.
What is being missed
Another use of code coverage over and above isolated values is the ability to look at what is not being tested. Don’t look at the green; look at the red. The red areas tell us what parts of the software we’re missing. We can look at e.g. class or method names and decide whether we can design tests that should cover these areas.
When a new test is added we can check whether it covers the areas we thought it would. Instead of just looking at a simple number, we should look at the details.
New questions
As it turns out, the questions about effectiveness and risk can’t really be answered by code coverage. What it can answer though, is "where are we not performing any testing?", "where are we performing too little testing?" and "what effects have my last changes (more/less code, more/less tests) had on the trend?". Based on the answers to these questions, we can design new tests. Code coverage can certainly help us to get better quality software, but perhaps not in the way that is often imagined.
