“Defense in depth” testing strategy
There is nothing that we can call “excessive testing”. The more you test, the better it is.
When we get to know the first few Laws of Testing and understand that “Testing can never be complete”, we get into the mode of limiting our testing to the constraints. This often includes selecting the best tests to execute in given time, leaving out some testing types and worse just having breadth in testing for the sake of leaving all the depth in testing.
Last week, I was discussing with my manager on the possible approaches we can take to add tests for multi-threading in our unit testing suite. He is based in the US, leads a big development team and has done many successful releases of complex systems, all approaches we discussed had some flaws and we couldn’t agree on the “best approach”. You can guess the decision we made. No, it was not to drop this type of testing altogether because we don’t know the best way to do that. The decision was to try the approach that is one of the candidates of becoming a good approach, do it for a while and learn from it.
That is what my manager described as a test strategy to have “Defense in depth” where you can imagine a castle which is built on the top of a hill with a big wall and then outside fence and then there are trenches and then there are landmines around it and then there are other ways to protect the castle. All of these layers of defense have some advantage over the other and though they cannot protect from all type of attacks, they still limit the chance of attack by capturing some of it.
Or you can imagine a soccer team that has the Italian strategy to never allow the opposition to score a goal.
(the image is taken from here: http://blog.smartbear.com/design/what-medieval-castles-can-teach-you-about-web-security/ )
I have often shared my views about Unit tests being a safety net which allows Programmers to make changes without fear that if they fall, they’ll not fall on the ground. So in that respect, why just have one safety net, why not have multiple nets such that if the fall is big and it raptures the first net, the other one saves you from falling on the ground.
In my experience, the following layers of protection or safety nets are kind of a must:
- Set of Unit tests that run with the individual code manipulations. They are by default automated and usually executed multiple times in a day.
- Set of tests that are executed immediately after the build is made. These are Smoke tests and other tests such as Performance which were not executed as part of the build. These should also be automated tests and run with each build.
- Set of tests that are Exploratory in nature and are executed by Testers who have knowledge of the domain and the overall system. Some experts believe that these tests are useless or repetition if we have the above two layers but as I mentioned above, they add another layer of defense. And I have seen many big bugs coming from this round of testing as the above two are merely Regression tests.
Do you believe in having depth in your software testing layers? How many layers you have or would like to have?