In the game of Numbers

In the game of Numbers, some numbers are highlighted in a way that other numbers do not get attention. Governments do that every day, CEOs do that every quarter in front of board of directors, Scientists stumble upon this in every research and software practitioners exercise this every release cycle.

Aaron Levenstein rightly said

“Statistics are like bikinis. What they reveal is suggestive, but what they conceal is vital.”

Let’s see how the game is played.

Number of Defects

This is by far the most favorite number for all: Testers, Programmers, Project Managers etc. etc. Testers use this to show how buggy is the software by saying “there are 254 open defects” where as Programmers use this as “there are 84 defects yet to be verified by testing team”. We see graphs showing the trends of incoming defects, outgoing defects. We sit in discussions where fancy charts are shown for the status breakup of the defects. All, but we forget that the number of defects is not the only measure of quality until we know what risks they expose.

One game that I’m playing right now is that recently we got 300+ defects to be verified at once due to some odd reason of scheduling. Now as a sensible tester, I should be testing the most critical defects or the defects that came directly from clients or the defects that expose the biggest risk. But since we like playing the game of the numbers, I picked the easy ones such that defect number each day is 275, 240, and so on.


(the original photo is here:

Number of Tests

This is the favorite number of Programmers, “hey we have 15,000 tests running with each build”. Third party tools market this to prove that their tool is of high quality.

One game that is played here is to quickly bump up the number by writing similar tests. On one project that I worked, one guy added 1200 tests in a week. Your guess is right that they were value parameterized tests checking the same feature using a variety of data. Not that this variation is not important but saying that we have thousands of tests in not the only measure of quality until we know what those tests do.

Number of Test Cases

This is sophisticated technique deployed by experienced testers to delay the release and we hear statements like “there are still 112 test cases which are not executed yet”. These test cases are decorated in online repositories, central storages and often come up with some beautiful status diagrams.

One game that is played here is predicting the release. “Given that we have 700 test cases that take about 800 hours of testing and counting a 40 hours week, we need more testing time than scheduled”. Another one is “until the last test case is not executed, I cannot authorize the release”. The Regression of all those test cases is all that important, but number of remaining test cases is not the only measure of quality until we know what is remaining.

The real game

I think we need to stop playing these games. These numbers should be used in the right context and should be made meaningful.

And this is our opportunity to train our teams as James Bach and Aaron Hodder wrote that:

“All metrics that are used by management to control people is used by the people to control management”

Do you play this game of numbers? And you have an exit plan?

Tags: , ,

14 responses to “In the game of Numbers”

  1. Farrukh Latif says :

    Very relevant and practical phenomenon, a tool for most of the fellows 🙂


  2. rumadak says :

    So true! sometimes collecting these metrics cause more wastage of time than the actual benefits they provide, if any.

    As for your question, moving to Agile might be a way, as there is not much time for unnecessary metrics.
    But, it largely depends on your organisation, team, etc etc.


    • majd says :

      Thanks Ruma and Agile helps here. But obsession to metrics is also observed in Scrum Masters at least in some with whom I have worked. Though true Agility would ask questions about those metrics and then provide a better picture.


  3. H Hamid says :

    Good share


  4. ahmedmugheera says :

    Reminds me of SAW: I want to play a game!
    Here’s an exit plan. Let’s play with some other, more serious numbers 🙂 This is an unfair world where everything eventually translates to cold hard money: numbers 🙂
    Suppose your game of numbers has concluded somehow and the product is finally ready to roll. I invite you to look at the other (shall I say gruesome?) side of the picture. Numbers – after the team is done with development and testing! Testers and Bugs only 🙂

    Recently, I am experiencing some new metrics:
    First, number of defects found by customers (after a release). Here, this directly relates to the concept of ownership, accountability and performance measurement, and a (somewhat painful) way to let testers’ improve their planning and effectiveness in next releases.

    Another interesting (and a bit cumbersome) metric: Number of bugs logged on release-day. Here, it’s an all-hands testing saga during and just after the release. Any unanticipated issues, and testers get the heat. Again, it lets us improve ownership, risk-assessment and escalation aptitude, and plan how to be more effective.

    Another one which is slightly less critical: bugs being logged after the regression. Again, it’s about the quality of regular testing done during the development sprints, and again it is attributed towards improving testers’ skills.
    As difficult or awkward as these may sound, I think these are some useful and meaningful metrics, that can be used as effective feedback.

    You are right that some numbers are irrelevant, and others less important, and that we should go after the most meaningful ones. Some of them matter less and some more, but numbers matter 🙂


    • majd says :

      Thanks Ahmed for detail analysis on the subject. Indicators like bugs found after release, on last day are good insight into future projects and improving test strategies/minimizing risks. But if these are used for ‘blame game’ then that is worse than ‘number game’ 🙂


  5. shahidullah khan says :

    Good share. I think the better approach is to focus on Test scenarios rather then test cases. In this way we can avoid this ‘number game’ 🙂


  6. Rebbeca says :

    This assists you the absolute most amount of cash over the long term.


Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s