Code Coverage is a good indicator of how well your tests are covering your code base. It is done through tools through which you can run your existing set of tests and then get reports on code coverage with file level, statement or line level, class and function level and branch level details.
Since the code coverage report gives you a number, the game of numbers kicks in just like any other number game. If you set hard targets, people would like to get it, and at times a number means nothing. Here are my opinions based upon experience on how to best use Code Coverage in your project:
Do run code coverage every now and then to guide new unit test development
It’s worth running coverage tools every so often and looking at these bits of untested code. Do they worry you that they aren’t being tested?
The question is not to get the list of untested code base; the question is whether we should write tests for that untested code base?
In our project, we measure code coverage for functions and spit out list of functions that are not tested. The testing team then do not order the developers to write tests against them. The testing team simply suggests to write tests and the owner of that code prioritize that task based upon how critical is that piece and how often that is requested by the Users.
Dorothy Graham suggests in this excellent talk that coverage can be either like “butter” or like “strawberry jam” on your bread. You decide if you want “butter’ like coverage i.e. cover all areas or you want “strawberry jam” coverage i.e. cover some areas more in depth.
Do not set a target of 100% code coverage
Setting up a coverage goal is in itself disputed and is often misused as Brian Marick notes in this paper which has been foundation of any Code Coverage work thereafter. Also anything that claims 100% is suspicious e.g. consider following statements:
- We can’t ship unless we have 100% code coverage
- We want 100% reported defects to be addressed in this release
- We want 100% tests to be executed in each build.
You can easily see that a 100% code coverage gives you “Test all Fallacy” to imply that we can test it all. Brian suggests in the same paper that 90 or 95% coverage is good enough.
We have set a target of 90% function coverage but it is not mandatory for release. We provide this information on the table along with other testing data like results of tests, occurrence of bugs per area etc. and then leave the decision to ship on the person who is responsible. Remember, the job of testing is to provide information not make release decisions.
Yes, there is no simple answer to how much code coverage we need. Read this for amusement and know why we get different answers to this question.
Do some analysis on the code coverage numbers
As numbers can mean different things to different people, so we need to ask stakeholders why they need code coverage numbers and what they mean when they want to be covered.
We asked this question, got the answer which is to do a test heat analysis on our code coverage numbers. It gives us following information:
- Which pieces are hard to be automated? Or easy to be automated
- Which pieces are to be tested next? (as stated in first Do)
- Which pieces need more manual testing?
- How much effort is needed for Unit testing?
Do use tools
There are language and technology specific tools. For our C++ API, we have successfully used Coverage Validator (licensed but very rightly priced) and OpenCppCoverage (free tool) that extract info by executing GoogleTest tests.
Do not assume coverage as tested well
You can easily write a test to cover each function or each statement, without testing it well or even without testing it at all.
Along with our function wise code coverage that I mentioned above, we have a strong code review policy which includes reviewing the test code. Also we write many scenario level tests that do not add to the coverage but cover the workflows (or the orders in which functions will be called) which are more important to our users.
Brian summarizes it nicely in the before mentioned paper:
I wouldn’t have written four coverage tools if I didn’t think they’re helpful. But they’re only helpful if they’re used to enhance thought, not replace it.
How you have used Code Coverage in your projects? What Dos and Don’ts you’d like to share?
If there were a time to test the passion of Software Testing community in Pakistan, it was last Saturday when resilient members attended the 6th Islamabad Testers Meetup in big number despite all the traffic challenges. The event hosted by MTBC at their campus in collaboration with PSTB saw professionals of various backgrounds who discussed and learned lot of new ideas on the day.
The event was a much awaited one as after the successful PSQC’17, activities restarted to gear towards PSQC’18. Umer Saleem welcomed everyone in the morning as Host of the day, introduced the event agenda and handed over the stage for the first talk.
Kalim Ahmed Riaz, Senior SQA Engineer from Global Share presented his thoughts on “Ways to Improve Software Quality”. Kalim who maintains his own blog on testing, quoted several everyday examples to explain his views including buying vegetables and the knowledge needed to define requirements. He then suggested multiple ways to cure common Quality diseases including “treating Testers as Clients”. By which he meant that not only Testers need to step up to think and act like real Clients but also the management need to acknowledge issues reported by Testing team to be real one and not postponing them sighting they are not real users. His slides are here: IsbTestersMeetup_Ways_improve_Quality_Kalim
The next presentation was well handled by a Story-teller Farrukh Sheikh who is Lead QA Engineer at Ciklum. Farrukh shared his thoughts on “Adaptation” skill which he believes is a must for all testers. He emphasized everyone to get out of their “deceptive illusion” where they believe themselves as strong and well-built testers and rather learn the new skills of the trade. He suggested to read a lot, practice new methodologies, learn coding as few ways to build muscles for tackling the challenges of tomorrow. Full slide deck: IsbTestersMeetup_Adaptation_Farrukh
(more photos are here)
After the talks, we did an experiment of “Open House Discussion” which was very well received. I was handed over the task to moderate it around the selected topic (based upon registered participant’s feedback) “Challenges in Implementing Test automation”. The format was that anyone from audience would share a challenge he or she is facing, and then everyone in the audience was free to give input on how to address that challenge. The discussion was around following challenges shared by members present there:
- Not finding time to do automation
- How to stabilize automation when everything is changing?
- Automation falling at configurations or keeping them up-to-date
- What’s next if first round automation is done
There were many suggestions offered from experienced and uniquely qualified testers in the hall which ranged from “talking to management to buy time”, “involving Programmers in the team to be part of the automation project”, “including automation tasks in Sprint planning”, “keeping an open eye on the changes within and outside your organization”, “being selective in what to automate” and so on. The discussion was of much worth and I plan to write a full blog post on these topics. It was heart-warming to see that we as a community are now moving forward to discuss such challenges and hopefully next time we will have solved these problems and ready for next set of problems.
Dr. Zohaib Iqbal, President PSTB was then invited to give updates from PSTB. He shared the lessons learned so far in holding such events and highlighted various programs including Certifications around CTFL, ISTBQ partner program and Accredited training program. He invited all to be part of upcoming PSQC’18 to be happening at Lahore in Spring next year.
The stage was then handed over to Adeel Sarwar, CTO MTBC for a closing note. Adeel briefly talked about MTBC vision including their wonderful initiatives to incorporate fresh college graduates into their workforce and how this model has worked really well for them. He raised his concern of people in IT not reading enough which reminded me of my campaign of “Certified Book Reading Tester”. He made it clear to everyone that we need to raise the bar to improve quality of Software Quality professionals. He thanked the audience and shared some ideas on future collaborations in which MTBC can be a part.
Shields were presented to speakers and certificates were given to the wonderful MTBC organization team who were well applauded by round of applauses. We then moved to outside for an open air tea with snacks setting. The food was as delicious as were the discussions and lot of new ideas were shared by participants. New friends were made and old friendships were revived. The conversations were never ending but with every beautiful meeting, it ended too soon for all of us.
Thanks again to the hosts, PSTB support and above all the “cheetah” testers who always respond to such events. Together, let’s make Pakistan testing community even stronger!
Jerry Weinberg is my favorite author on the subject of Software Testing and I have posted about lessons from his Errors book, shared my thoughts on some types of Bugging. Lately I was very happy to finish yet another book by him called “Perfect Software and other illusions about Software Testing” which was on my reading list for log due to excellent reviews.
Now having spent almost 15 years on Software Testing domain and overall 20 years in the pursuit of Software Quality in general, my first impression was that the illusions Jerry will talk about will be one of those that I’m familiar with and/or have dealt with. But Jerry surprised me with his wealth of knowledge and interesting examples from his illustrious journey as a Software professional. I learned lot of lessons and sharing the top 3 with you.
- Pinpointing problems is not Testing
Some activities that I always thought to be Testing’s responsibility are actually in a grey area where these have to be shared between Testers and Programmers. Pinpointing a problem, after it is found, is one of them.
In one of the early team that I worked with required that when Testing reports a bug in the Tracking System, they should mention the first major version it was introduced. For example, if a Tester finds a bug in v 3.4, Tester would check if it existed in v 2.0 and v1.0 (if the answer to 2.0 is no). That is an activity that Jerry calls Pinpointing and suggests to differentiate it from Testing and Debugging.
Even in my current project, because we (the Testers) have the code and are able to debug an issue, our Data conversion issues are reported after pinpointing and debugging. Hence a Tester’s considerable time goes into locating the problem code after finding the problem. This is included in Testing whereas it needs some separate attention.
In Jerry’ words as he summarize this as a Common Mistake and suggests a solution:
Demanding that Testers locate every fault: This is totally a Developer’s job, because developers have the needed skills. Testers generally don’t have these skills, though at times, they may have useful hints.
- Providing Information is hard
Jerry talks in much detail Tester’s role as information provider. He suggests to use Satir interaction Model:
-> intake -> meaning -> significance -> response ->
There are separate chapters dedicated on each of the above and I’d encourage any Tester who is serious about their job to understand this model through reading the book. Different techniques have been discussed to improve information intake, how different people will mean different things from same observation, how to know which information is significant and finally how to make the best response accordingly.
- Product reviews are actually testing
In fact Jerry calls it “Testing without Machinery” and suggests that technical product reviews are actually a way to provide information and thus is Testing. He also lists some “instant Reviews” which is that if you try to review and you hear excuses like below, you know that something is wrong:
- It’s too early to review it.
- It’s too small to bother with. What could possibly go wrong?
- We’ll let you know as soon as we are ready.
- We [tested][tried][demonstrated] it and it works okay.
It’s a treasure of information on those few pages which you can get instantly once you ask for a review. This has inspired me to write more about Peer Code Reviews so you should expect an article soon.
Let me end this with Jerry’s words:
If humans were prefect thinkers, we wouldn’t need to test our work. … But we are imperfect, irrational, value-driven, diverse human beings. Therefore, we test, and we test our testing.
“In Pakistan, testers meet up happen rarely and if it happens, it must be appreciated. Stella Technology organized workshop named as ‘Insight into Software Security’ last Saturday i.e. Sep 23, 2017 in their Islamabad office. Main objective of workshop was to provide testers community a space to share knowledge about Software Security.
Continue reading at original post here: https://cymakarim.wordpress.com/2017/09/26/insight-into-software-security-workshop/
Test execution is perhaps the most neglected area in the field of Software Testing. Too much focus is put on the test planning and design part as compared to test execution and reporting. One reason for this is that test planning/design is considered to be the intellectual part of the equation where the skills of a Tester make the most difference.
Let me try to take this topic today.
The first and foremost objective of test management for the execution phase is to ensure that all tests that have been written are executed on regular basis. The initial investment of writing Unit tests or any automated tests is paid off when those test execute for may be 1,000 times. You should always remember:
A single test that is executed thousand times is better than one thousand tests that are never executed.
You may be saying that how come a test is written but it is not executed. But it’s true. Tests get hidden in the system. At times, they are put in the ignored list to have your build passing, and no one looks at them again. At times, they are executed on only one configuration and other important configurations go missing due to either lack of resources or due to poor planning.
The next thing to consider is how much time it takes to execute them. Because if you leave things as it is on the test execution side, your testing starts becoming expensive. The old saying “time is money” is coming ever true with hardware and software being acquired as a service and paid on per hour basis. The more time you spend on test execution, the expensive it becomes.
If your test execution is all manual, it is by any standard the most expensive part of the system. And if some of it is automated, keep an eye on how much time it take to run. It may be fast but it has to be super fast.
For example in one of the projects that I’m working, we have over 5,000 unit tests that are executed with each Developer build. The overall execution time reached to 25 minutes which was even more than the actual build time. We parallelized it and time got reduced to 15 minutes. Now we are working at optimizing test startup time and data creation to even reduce the time. The saving is huge as it cuts the time for each Developer and enhances the overall Developer productivity.
Next thing to consider is how frequent your test run. If you run your Performance tests once in a week, then even if you notice a dip in performance it is really hard to pin point the actual commit message that caused the Performance to go down. If you can move it to daily, your chances suddenly are better to trace back such issues. Luckily many automation server tools help you orchestrate your jobs and you should view tools like Jenkins not only as Continuous Integration tools but as Continuous Testing tools.
The final comment on test execution is how the results are being reported. Most test execution results are raw and are in the shape of data. One thing that we did recently is to write series of Python scripts to manipulate that data and transform that into information. Such that results of our test execution are reported immediately on a Testing Dashboard in a Management friendly way like testing heat analysis . The end result is that at any time a Project Manager (or Product Manager or whoever is responsible for release decisions) can go and see what is the risk if we release just now.
How much emphasis you give to test execution phase? Have you tried something that is worth sharing with others?
Performance testing is one of the specialty area for testers and is one of the challenging tasks. Designing Performance tests is different than designing functional tests, then setting up testing environment for Performance also needs special care and finally figuring out if the results do indicate an issue is not straight forward. One thing that has helped us lately in troubleshooting Performance issue is logging.
It is a known fact that logging is a Tester’s friend. Rather I’d say that it is a friend for any troubleshooter. The logs are like the traces which a skilled investigator follows to find the culprit who committed the crime. These are “Khurras” (Punjabi meaning foot prints) which “Khojis” (Punjabi/Urdu/Hindi meaning Detectors) follow. If the crime was committed (read an issue was observed) but no “khurras” are there (read no logs are there), it is not possible for the “Khoji” to find out the criminal (read what is the root cause of the issue).
(Original photo is here)
The first question in setting up logs is what logs we need? These have to be key points in software behavior for which time/memory captures are added to the log. For example in one of our Display systems where we implemented logging, we added logging for the following:
- The time to open the file.
- The time to display the thumbnails in the file.
- The time to fill view with selected thumbnail.
- The time to show properties when an element is selected in the display.
Once you know what points you need to log, the next step is to hook in logging. I’m hoping that your software or the tools/technologies that is being used by your software already has logging infrastructure in it and you have to add a new logger for Performance. And if it is unfortunately not there, then you need to have such infrastructure first. This logging will be added by the Programmers in your team and thus this project is another example of how Programmers and Testers can work together to deliver quality.
Suppose you have those friendly Programmers in your team and Performance logging is enabled. Now all you have to do is turn it on and run your existing tests. That’s it. Along with the numbers that measured before, you’ll get the logs.
Remember when all is going good, no one needs logs.
So you’ll need the logs only when your results are indicated a decline in Performance numbers that you measure. And then you can pass it on logs directly to the Programmers to investigate. Or if you are a smart tester like me, you’d write a small Python script to parse the log and get the meaningful numbers that you report to your team.
We’ve tried the above and believe it is really helping in issue investigation. We have moved through this from “We have poor Performance numbers” to “The Performance is down because file goes through up gradation progress while opening” like concrete issues being reported and fixed. Try it and you’ll love it. It will take some effort from both the Programmers and Testers to do it but it is surely worth an effort if you are serious about Performance testing.
Do you use logging in Performance testing? Or you have some other interesting ways to investigate Performance issues?
Last week I was invited to speak to graduating batch of BS Computer Science at FAST Islamabad Campus. This was part of an initiative through which Industry professionals visit students and give them a true picture of real world. On my previous visits, I have talked about different aspects of Software Testing whereas this time I picked up misconceptions.
There is a long list of misconceptions but here were the 3 that I talked about:
- Don’t like Coding; Become a Tester.
That’s number one as many in Academia still believe Testing to be a non-coding job. That may have been true in 1990s when the profession of Software Testing was new and the gap was filled by people from other professions who were expert users and became testers. But the latest job market requires more and more testers who can write code to test code.
- Testers are paid less than Programmers.
This may have come in due to earlier practice of Industry to pay less to Testers who were Clickers in some respect. The pay scale of job reflects its complexity and it was fair to pay dumb testers less than Programmers. But again as the recent demand of smart and knowledgeable testers who are in some ways Gray Box Testers, the pay scale need to match that of a Programmer as both roles require intellectual work.
- Testers don’t grow. All important positions go to Programmers.
With testing becoming central to modern day Apps, testers are growing in multiple directions. There are testers who grow to run the entire testing department of the organizations. There are testers who grow to lead Engineering departments. And there are testers who become Testing experts in Performance testing or Automation or some other domain.
I used the Speakers list from our PSQC’17 that was held last month to give examples from each category. That really helped because local examples work more than any thing and I realized that holding a Software Quality Conference is helpful in many ways.
In the end there were many questions around the above myths and the realities involved. One question was aimed towards bitter relationship between Testers and Programmers for which I answered that this behavior needs to go and we need friendship between them to be able to deliver Quality Software.
What else you would like to add to this list of misconceptions?