Test Execution in focus
Test execution is perhaps the most neglected area in the field of Software Testing. Too much focus is put on the test planning and design part as compared to test execution and reporting. One reason for this is that test planning/design is considered to be the intellectual part of the equation where the skills of a Tester make the most difference.
Let me try to take this topic today.
The first and foremost objective of test management for the execution phase is to ensure that all tests that have been written are executed on regular basis. The initial investment of writing Unit tests or any automated tests is paid off when those test execute for may be 1,000 times. You should always remember:
A single test that is executed thousand times is better than one thousand tests that are never executed.
You may be saying that how come a test is written but it is not executed. But it’s true. Tests get hidden in the system. At times, they are put in the ignored list to have your build passing, and no one looks at them again. At times, they are executed on only one configuration and other important configurations go missing due to either lack of resources or due to poor planning.
The next thing to consider is how much time it takes to execute them. Because if you leave things as it is on the test execution side, your testing starts becoming expensive. The old saying “time is money” is coming ever true with hardware and software being acquired as a service and paid on per hour basis. The more time you spend on test execution, the expensive it becomes.
If your test execution is all manual, it is by any standard the most expensive part of the system. And if some of it is automated, keep an eye on how much time it take to run. It may be fast but it has to be super fast.
For example in one of the projects that I’m working, we have over 5,000 unit tests that are executed with each Developer build. The overall execution time reached to 25 minutes which was even more than the actual build time. We parallelized it and time got reduced to 15 minutes. Now we are working at optimizing test startup time and data creation to even reduce the time. The saving is huge as it cuts the time for each Developer and enhances the overall Developer productivity.
Next thing to consider is how frequent your test run. If you run your Performance tests once in a week, then even if you notice a dip in performance it is really hard to pin point the actual commit message that caused the Performance to go down. If you can move it to daily, your chances suddenly are better to trace back such issues. Luckily many automation server tools help you orchestrate your jobs and you should view tools like Jenkins not only as Continuous Integration tools but as Continuous Testing tools.
The final comment on test execution is how the results are being reported. Most test execution results are raw and are in the shape of data. One thing that we did recently is to write series of Python scripts to manipulate that data and transform that into information. Such that results of our test execution are reported immediately on a Testing Dashboard in a Management friendly way like testing heat analysis . The end result is that at any time a Project Manager (or Product Manager or whoever is responsible for release decisions) can go and see what is the risk if we release just now.
How much emphasis you give to test execution phase? Have you tried something that is worth sharing with others?