Management friendly test reports
The Manager Responsible for Release (let’s call that person as MRR for rest of this post) has to make the decision to ship the product and is keenly watching the Iteration Review. Programmers have finished demoing the implemented features and the situation looks good so far. The testers start their part and this is what MRR get:
“14 new bugs reported in the Iteration out of which 8 were of highest severity. 26 were verified and closed. There are still 4 open Defects.
Out of 18 test plans executed, 13 are passing and 4 are failing with some issues reported while 1 is still to be executed.
Smoke testing was done for each of the 4 builds in Iteration and 3 passed whereas 1 failed.
Performance tests were executed on the Release Candidate build.
That’s all from the testing team.”
Confused as MRR wants to make a decision, the person un-mutes the mic on the conference call and asks: “Tell me if we are good to release the build demoed earlier in the Review?”
The test lead says: “That decision is not Testing’s rather it is yours. But if you ask us, we say may be.”
The MRR loses confidence in the testing team in particular and in testing in general (along with losing the temper).
Does the above scene look familiar to you? That a testing status is entirely a report on the testing activities with too much focus on numbers? And do as testers, we need to change our tactics? I think yes.
What do testers do? They provide information about product under test, to expose risks for the team.
In the above example scenario, does the tester provide information that exposes risks? Obviously not. Let’s revisit it and see how activity report can be used to do this.
What is the risk?
Before you prepare your testing report, you should be very clear on what risks are there in management’s mind. These could be in the line of following statements:
- What if the product it not released on time?
- What if the product is not stable enough to be released?
- What if the product is too slow to use?
What information Management needs about that risk?
Management is looking for information in the report and not data (data vs. information). Let’s convert the earlier data and convert it into information:
“There are 4 open defects and they are all related to View Profile feature. Given that this is our first release where Users will mostly be working with their Profiles, they possess a threat to the release if not fixed. 1 defect is of high severity which allows to view profile of any user through modified URLs and fixing it is of high value but of not much cost. The other 3 defects are related to some cosmetics which are important but can be let go if needed.”
Now the MRR feels much informed and says: “Great. If we can fix that and do a round of regression, we should be able to ship in 2 days”. All say yes and life is good (at least for now).
Can you think of similar statement for the test plan results or smoke test or performance results in above example?