Is your testing well spread?
Testing is in some ways similar to medical examination. Apparently things look okay but a first pass tells something may be wrong under the scene and a thorough investigation is suggested. Those detailed tests then reveal if the “system under test” or “body under examination” is fit or not for the intended purpose.
That’s why the set of tests in your testing strategy should be well spread.
Take smoke testing for example. Smoke testing tries to cover as many as surface area of the system rather than going into in-depth analysis. A typical example is that if your application has 7 menu items and you have 7 test plans focused on each menu item, your smoke test might be something that calls all the 7 menu items to see it all works fine.
During an office wide presentation on “Unit Testing” that me and a Programmer colleague of mine did at our office a few weeks ago, the Programmer friend explained the following model to explain the concept of well spread Unit Testing.
The programmer in Exhibit A is obsessed with a specific area of the Unit. This could be due to multiple reasons including a) Programmer thinks that this is the most important piece that outside world is interested, b) Programmer knows that area well, c) A bug was reported for that area and so on. But this approach is not good as it misses out most of the surface area. Exhibit B is what should be sought after and as soon as you write some tests about a specific area, you should plan to move to other surface areas.
This well-spread testing strategy is also apparent in the Agile Testing Quadrants where this model helps you see how your testing effort is concentrated. If a team has too much focus on Programmer written technology facing tests and forgets the Testers doing Business facing testing, there will be risks to the project and vice versa. Similarly I have seen teams leave Performance testing towards the end of projects where those findings can’t help much as changes to technology or design is not possible at that stage. Performance testing should be done early in the project to help team pick the best suited technology and lay foundation of the design in the right direction.
Another way to look at this problem is through testing heat analysis that I presented sometime ago to help decide where to put the next tester. The areas that have no testing are very vulnerable because the only truth comes when you test the system. Everything else is a wild guess of how things might be working.
How’s your testing efforts distributed? Like Exhibit A above or Exhibit B and what suggestions you have based on your experience.