Code Coverage Dos and Don’ts
Code Coverage is a good indicator of how well your tests are covering your code base. It is done through tools through which you can run your existing set of tests and then get reports on code coverage with file level, statement or line level, class and function level and branch level details.
Since the code coverage report gives you a number, the game of numbers kicks in just like any other number game. If you set hard targets, people would like to get it, and at times a number means nothing. Here are my opinions based upon experience on how to best use Code Coverage in your project:
Do run code coverage every now and then to guide new unit test development
It’s worth running coverage tools every so often and looking at these bits of untested code. Do they worry you that they aren’t being tested?
The question is not to get the list of untested code base; the question is whether we should write tests for that untested code base?
In our project, we measure code coverage for functions and spit out list of functions that are not tested. The testing team then do not order the developers to write tests against them. The testing team simply suggests to write tests and the owner of that code prioritize that task based upon how critical is that piece and how often that is requested by the Users.
Dorothy Graham suggests in this excellent talk that coverage can be either like “butter” or like “strawberry jam” on your bread. You decide if you want “butter’ like coverage i.e. cover all areas or you want “strawberry jam” coverage i.e. cover some areas more in depth.
Do not set a target of 100% code coverage
Setting up a coverage goal is in itself disputed and is often misused as Brian Marick notes in this paper which has been foundation of any Code Coverage work thereafter. Also anything that claims 100% is suspicious e.g. consider following statements:
- We can’t ship unless we have 100% code coverage
- We want 100% reported defects to be addressed in this release
- We want 100% tests to be executed in each build.
You can easily see that a 100% code coverage gives you “Test all Fallacy” to imply that we can test it all. Brian suggests in the same paper that 90 or 95% coverage is good enough.
We have set a target of 90% function coverage but it is not mandatory for release. We provide this information on the table along with other testing data like results of tests, occurrence of bugs per area etc. and then leave the decision to ship on the person who is responsible. Remember, the job of testing is to provide information not make release decisions.
Yes, there is no simple answer to how much code coverage we need. Read this for amusement and know why we get different answers to this question.
Do some analysis on the code coverage numbers
As numbers can mean different things to different people, so we need to ask stakeholders why they need code coverage numbers and what they mean when they want to be covered.
We asked this question, got the answer which is to do a test heat analysis on our code coverage numbers. It gives us following information:
- Which pieces are hard to be automated? Or easy to be automated
- Which pieces are to be tested next? (as stated in first Do)
- Which pieces need more manual testing?
- How much effort is needed for Unit testing?
Do use tools
There are language and technology specific tools. For our C++ API, we have successfully used Coverage Validator (licensed but very rightly priced) and OpenCppCoverage (free tool) that extract info by executing GoogleTest tests.
Do not assume coverage as tested well
You can easily write a test to cover each function or each statement, without testing it well or even without testing it at all.
Along with our function wise code coverage that I mentioned above, we have a strong code review policy which includes reviewing the test code. Also we write many scenario level tests that do not add to the coverage but cover the workflows (or the orders in which functions will be called) which are more important to our users.
Brian summarizes it nicely in the before mentioned paper:
I wouldn’t have written four coverage tools if I didn’t think they’re helpful. But they’re only helpful if they’re used to enhance thought, not replace it.
How you have used Code Coverage in your projects? What Dos and Don’ts you’d like to share?