The best answers to “what is the job of testing?” always have a key component suggesting that it is “to provide information”.
Testers job is to provide information for the product under test
They provide information about product under test, to expose risks for the team.
Now if you have been provided information any time in your recent past, you can easily recall that some of it was valuable and some of it was not. Even some information which was provided to you was false, misleading, hiding the facts or inflicted with similar diseases. So how do we, the testers, can provide information which is always always valuable to the person who is receiving information?
I believe the answer is in two parts: a) How do we collect information? and b) How do we process and present the information?
May be I am bit vague above, so let me spend some time to explain myself.
To provide information, you need to have all the information available to you collected from different sources like test results, design documents, release schedules etc. And then you need to process this information for the given need like the occasion, the person or timing of the information being provided. Like providing specific filters to view information from a certain angle.
Let’s dig a bit deeper.
How do we collect information?
Some testers are overly relying on only the results of testing. They don’t like to discuss with Development about how those tests are designed or how the results are interpreted rather they believe that results of testing are the only source of truth.
Then there are others who love documents. All of the information they have is primarily coming from a document e.g. Design document, Project Plan, Release Plan, or some sort of change tracking system like TFS. For them, those documents are the only source of truth.
And there are the third group who like verbal communication. They discuss each and every type of test in daily SCRUM calls and get feedback on Iteration Review results. They discuss release plans with the management rather than looking at what the document suggested at the start. They are of the view that people involved in the team are the only source of truth.
Not many people know that if all the information you have is coming from a single source, you don’t have a complete view. Consider political arguments that happen in your work environment between people who watch/follow a particular news channel.
So if as a Tester, you want to improve your skill as a better information provider, the first step is to widen your input sources. Test Results, Documents, People all matter. And even there are other sources which you know better than me.
How do we process and present information?
So when you are asked “How was your weekend?”, your answer largely depends on who is asking and in what circumstances. Your answer will vary in details and emotions for your jogging partner compared to your colleague; your best friend compared to someone you are meeting for the first time at a business meeting.
Similarly when you are asked “How is the testing going?” or “Are we release ready?”, if you are giving the same answer to every person in every possible settings, it shows that you are not filtering the information you are having and passing it on as it is. Apply different views to the same information given the context and your answers will contain valuable information for the receivers.
What tips you have to improve information gathering and processing steps?
KnowledgeTester turned 5 few days ago!
And as I discussed the journey in first, second, third and fourth year, my first happiness is to see that it is still going. I am, just like many of you, type of a person who “has a habit of starting projects and not completing them”. Though the project of blogging never finishes but continuing it for 5 years certainly can be used as a yardstick to call it mature.
(the original image is here: http://learnchineseinyunnan.com/news/news-wire/2015/08/20/2015-the-fifth-anniversary-of-thanks-giving/ )
The biggest achievement of fifth year is not a personal one but a collective one. And I sincerely hope you know that collective success has more savor than a personal one. And that was the first ever testing conference in Pakistan that we saw happening in April 2017 in a great way. My report on it “Pakistan Software Quality Conference 2017” has been the most viewed post of the year and I’m so happy to have achieved this through all friends specially the support of Pakistan Software Testing Board.
Interestingly that was my top most priority when I recapped fourth year. Now you can easily guess that this year’s priority is to make PSQC a regular thing in our Industry and we are working for PSQC’18 which will happen in February 2018 in Lahore Insha Allah (God Willing). Though we went into a momentary post conference enjoyment mode, where we made little progress in recent months but we are now getting back with two meetups coming soon in the remaining months of 2017.
On the viewership increase, I am seeing a 20% increase in per post views which may not seem great compared to previous years. But you know as the baseline increases, it becomes hard to keep increasing at same pace e.g. think how salary raise in % is tougher as your salary gets higher. But net effect is that views per post are almost at 1,000 now (967 to be precise) in last year. Yes, I can’t believe that this viewership is for the stuff that I’m writing but it’s true 😊
During the last two years or so, due to the community building activities KnowledgeTester was low on providing training to Testers. The good news is that I’m setting this as my second goal for the sixth year i.e. to hold Training along with continuing community activities. Wish me luck.
That’s about it for now and I’m thankful to each of you who contributed towards this success. And I’m sure you’ll be with me in next years of this journey.
Do you have any suggestions on what kind of training we should be offering this year?
“In Pakistan, testers meet up happen rarely and if it happens, it must be appreciated. Stella Technology organized workshop named as ‘Insight into Software Security’ last Saturday i.e. Sep 23, 2017 in their Islamabad office. Main objective of workshop was to provide testers community a space to share knowledge about Software Security.
Continue reading at original post here: https://cymakarim.wordpress.com/2017/09/26/insight-into-software-security-workshop/
Every plan is perfect until the execution happens. As a losing team’s coach once said: “We had a perfect plan on the paper. The only problem is that the match was played on the ground and not on the paper”.
So should we stop striving for better planning? Of course not. Rather we should learn from each execution so that our next plan is better than the previous ones. Taking a hint from this thinking, let me share 3 tips based upon execution failures to improve your test planning.
I have selected the tips based upon 3 layers of what I’m calling Agile test planning. Such that as a Tester, you are required to do “Release Planning” or asked to do an “Iteration Planning” or asked to do an “Individual Tester Planning”.
Let’s take an example of “Release Planning” first.
Many years ago my Employer decided to start offering solutions in the Mobile world. Before that we were only in the Desktop world. Our first solution was for Android and then we worked on a Platform that can produce Apps in Android, iOS or Windows. A year or so after this has happened, I was asked to do some test planning for a release meant for a Client in China. I ‘assumed’ that they’ll need mobile solution as it was our top priority then. So my plan started with Android and then iOS testing on Simulators and Devices.
The plan looked great. Our first iteration started and by that time, I had no interaction with the Client. Client started reporting a lot of bugs and all of them were reported in Windows Desktop environment. I learned that this Client is only interested in good old Windows Desktop solution and doesn’t care about mobile.
The lesson which I learned is my tip # 1 for this blog:
Always ask question rather than assuming things
Let’s now move to “Iteration Planning”.
In a project that I was part as the lead tester, the Product Owner complained that the number of new bugs reported towards the end of Iteration is far greater than the ones reported at the start. My first response was that as more things get ready for testing towards the end, more testing is done and hence more defects were found.
Then I gave it a deep thought by looking at the history of tests we were executed. One pattern was there that certain test plans produced higher number of defects (following the heuristic that Software Bugs are like Real bugs) and the test execution order was left on the tester to pick. Tester knew that this test plan will consume lot of time and energy, so they left these to the end.
The rule that we made afterwards is my tip # 2 for this blog:
Prioritize test execution based upon which tests will fail first
This article can help you get started on Risk Based Testing.
Let’s look at the last one called “Individual tester planning”.
When I was young and raw, I had very ambitious plans for myself assuming that I’ll get all things in my favor. I planned it 8 hours a day with no distractions. Eventually I couldn’t always finish the stuff that I committed at the start of Iteration.
With age and lot of experiences, I have become more realistic about my estimates and always consider it as a 6 hours day. I also account for unknown distractions like network outage, random meeting popping up, sick days and what not. It’s not that I give a pessimistic by including all those factors but I do keep margin for some of them.
The lesson that I learned is my tip # 3 for this blog:
A plan that doesn’t incorporate change is not a plan
What tips you can offer from your execution failures to improve test planning?
Test execution is perhaps the most neglected area in the field of Software Testing. Too much focus is put on the test planning and design part as compared to test execution and reporting. One reason for this is that test planning/design is considered to be the intellectual part of the equation where the skills of a Tester make the most difference.
Let me try to take this topic today.
The first and foremost objective of test management for the execution phase is to ensure that all tests that have been written are executed on regular basis. The initial investment of writing Unit tests or any automated tests is paid off when those test execute for may be 1,000 times. You should always remember:
A single test that is executed thousand times is better than one thousand tests that are never executed.
You may be saying that how come a test is written but it is not executed. But it’s true. Tests get hidden in the system. At times, they are put in the ignored list to have your build passing, and no one looks at them again. At times, they are executed on only one configuration and other important configurations go missing due to either lack of resources or due to poor planning.
The next thing to consider is how much time it takes to execute them. Because if you leave things as it is on the test execution side, your testing starts becoming expensive. The old saying “time is money” is coming ever true with hardware and software being acquired as a service and paid on per hour basis. The more time you spend on test execution, the expensive it becomes.
If your test execution is all manual, it is by any standard the most expensive part of the system. And if some of it is automated, keep an eye on how much time it take to run. It may be fast but it has to be super fast.
For example in one of the projects that I’m working, we have over 5,000 unit tests that are executed with each Developer build. The overall execution time reached to 25 minutes which was even more than the actual build time. We parallelized it and time got reduced to 15 minutes. Now we are working at optimizing test startup time and data creation to even reduce the time. The saving is huge as it cuts the time for each Developer and enhances the overall Developer productivity.
Next thing to consider is how frequent your test run. If you run your Performance tests once in a week, then even if you notice a dip in performance it is really hard to pin point the actual commit message that caused the Performance to go down. If you can move it to daily, your chances suddenly are better to trace back such issues. Luckily many automation server tools help you orchestrate your jobs and you should view tools like Jenkins not only as Continuous Integration tools but as Continuous Testing tools.
The final comment on test execution is how the results are being reported. Most test execution results are raw and are in the shape of data. One thing that we did recently is to write series of Python scripts to manipulate that data and transform that into information. Such that results of our test execution are reported immediately on a Testing Dashboard in a Management friendly way like testing heat analysis . The end result is that at any time a Project Manager (or Product Manager or whoever is responsible for release decisions) can go and see what is the risk if we release just now.
How much emphasis you give to test execution phase? Have you tried something that is worth sharing with others?
Summers offer exploration options. And exploration is always good for Testers.
Couple of weeks ago, I spent few days in the picturesque valley of Neelum in Azad Jammu Kashmir. The main attraction was to enjoy the beautiful scenes and flowing waters across the way. But being Tester for life, I had a second mission: find some bugs.
We did not have to wait long.
At the entry point where Tourist information was recorded as the road touches the Line of Control (a loose border between Pakistan and India for the disputed Kashmir region), we were given a brochure from the Tourist department. It was very helpful as it had distances, important phone numbers and places of interest. But my 12 yo daughter (who has accompanied me on previous bug hunting summer excursions) spotted an issue.
Sorry for this being in Urdu but it supposed to mean the “Meeting point of Neelum and Jehlum rivers”. The funny part was that by taking the ‘mmm’ sound off the second river, it meant “Meeting point of Neelum reiver and Stupidity”. Yes we stopped there to make that meet happen.
On our way beyond this meeting point was a long travel along the Neelum river. There is a lesson of Context in the name of the river. The same river is called “Kishan Ganga” by our Indian neighbors. The funny part is that at certain points, the River itself is the border. For example in the above picture, I’d be calling the river as “Neelum” and people across the river would be calling it “Kishan Ganga”. We can have a long discussion on what is the “correct” name of the river but reality is that it is one river with two names. How familiar is this situation when we have a word fight on certain terminology and that’s why understanding each other’s point of view matters a lot. For example in building a good relationship between Developers and Testers.
English being 3rd language in our region with every valley/region having it’s own language and then Urdu as national language, there are so many spelling mistakes to be caught in the journey. For example a hotel had numbered it’s multi-story building as following:
As you can imagine that with so much greenery out there, the possibility of finding the real bugs was very high. Some of them were unique in their textures, some were disguised and some were shouting out loud to be caught:
And yes your guess is right. Where there are bugs, there are bug hunters. Some of them were also ready for a photo-shoot:
There were many other spelling mistake bugs which I caught but I’m leaving them as they were level 0 or low severity bugs 😊. But there was one thing for which I couldn’t find a bug: the beautiful landscape that had all in one picture like the one below from Arrang Kel:
And oh yeah, before I go I’d like to mention a life philosophy lesson that I found on one of Rickshaws in Muzaffarbad which read:
Reading “Saans hay to Chance hay” meaning that if you are breathing, you have a chance to do something. We get bogged down by stressing office environment, project fatigues, overwork but remember how worthy it is still being alive. Because
Saans hay to Chance hay
Have you explored any new area in your Summers? What did you find worthy of sharing?
Consider the automatic machines you have at home as an example. You have a Washing machine to do a laundry on any day that you want. And then you have Dish washing machine that can clean your crockery at your will.
Aha… someone would say, here is some common stuff. You see both of them are machines that do “washing”, so why not we generalize it and make it one machine that will be able to “wash” either clothes or plates. They both use water and some sort of “detergent” to cleanse the items we put in it; they both use rinsing as a way to take off all the dirt etc.; and they both use air to dry the items once the washing is done.
If someone approaches you with the above plan, you should immediately recognize that you are being sold a “Phakki”. This can be an excellent plot of any piece of fiction from cartoons to movies but it has nothing to do with the real world. So if someone sells you a Generalized Test Automation solution, your response should be similar to the above plan.
(Charlie Chaplin in “Modern Times” movie. Full clip is here)
The common stuff in the case of washing machine can be components that you have in both machines, like an electric motor, water drainage system, air blowers and other similar pieces. But the function that each machine performs stays unique and thus requires a unique design and implementation.
So when you approach automation of your testing activities, you can have some common components that you use in different solutions, but each solution will have it’s unique needs. For example, over last couple of years we have written many scripts to automate the following testing activities:
- Smoke Testing
- Performance Testing
- Data Conversion Testing
- Compatibility Testing
- Cross Platform Testing
- Memory Leak Testing
All these scripts are specialized as they target one type of testing at a time. The common pieces are just the sorts of “Get the build to test”, “Gather the results” and “Notify about results”. The actual code that automates Performance Testing is a different beast than the one we have for Compatibility Testing.
We do have a lean type of Test Automation Framework. For other cases, there can be a framework that takes a General test automation tool as a starting point and delivers a specific set of specialized implementation for your product. In one of the talks at our regular meetups, my tester friend Adeel Shoukat gave an automation tip that was roughly “You’ll need to write lot of your own test routines using tools as you cannot use the tools right off the shelf to test your application”.
If someone tells you that here is a Generic Test Automation suite that will solve all your testing problems, you are being goofed.
Stay away from those pocket pickers.
How has been your journey towards automation? Have you ever seen or implemented a Generalized automation solution by any chance?