In the game of Numbers
In the game of Numbers, some numbers are highlighted in a way that other numbers do not get attention. Governments do that every day, CEOs do that every quarter in front of board of directors, Scientists stumble upon this in every research and software practitioners exercise this every release cycle.
Aaron Levenstein rightly said
“Statistics are like bikinis. What they reveal is suggestive, but what they conceal is vital.”
Let’s see how the game is played.
Number of Defects
This is by far the most favorite number for all: Testers, Programmers, Project Managers etc. etc. Testers use this to show how buggy is the software by saying “there are 254 open defects” where as Programmers use this as “there are 84 defects yet to be verified by testing team”. We see graphs showing the trends of incoming defects, outgoing defects. We sit in discussions where fancy charts are shown for the status breakup of the defects. All, but we forget that the number of defects is not the only measure of quality until we know what risks they expose.
One game that I’m playing right now is that recently we got 300+ defects to be verified at once due to some odd reason of scheduling. Now as a sensible tester, I should be testing the most critical defects or the defects that came directly from clients or the defects that expose the biggest risk. But since we like playing the game of the numbers, I picked the easy ones such that defect number each day is 275, 240, and so on.
(the original photo is here: http://www.frontpagemag.com/2014/dgreenfield/lefties-turn-on-nate-silver-over-global-warming/)
Number of Tests
This is the favorite number of Programmers, “hey we have 15,000 tests running with each build”. Third party tools market this to prove that their tool is of high quality.
One game that is played here is to quickly bump up the number by writing similar tests. On one project that I worked, one guy added 1200 tests in a week. Your guess is right that they were value parameterized tests checking the same feature using a variety of data. Not that this variation is not important but saying that we have thousands of tests in not the only measure of quality until we know what those tests do.
Number of Test Cases
This is sophisticated technique deployed by experienced testers to delay the release and we hear statements like “there are still 112 test cases which are not executed yet”. These test cases are decorated in online repositories, central storages and often come up with some beautiful status diagrams.
One game that is played here is predicting the release. “Given that we have 700 test cases that take about 800 hours of testing and counting a 40 hours week, we need more testing time than scheduled”. Another one is “until the last test case is not executed, I cannot authorize the release”. The Regression of all those test cases is all that important, but number of remaining test cases is not the only measure of quality until we know what is remaining.
The real game
I think we need to stop playing these games. These numbers should be used in the right context and should be made meaningful.
And this is our opportunity to train our teams as James Bach and Aaron Hodder wrote that:
“All metrics that are used by management to control people is used by the people to control management”
Do you play this game of numbers? And you have an exit plan?