3 Tips for near-perfect Test Planning

Every plan is perfect until the execution happens. As a losing team’s coach once said: “We had a perfect plan on the paper. The only problem is that the match was played on the ground and not on the paper”.

So should we stop striving for better planning? Of course not. Rather we should learn from each execution so that our next plan is better than the previous ones. Taking a hint from this thinking, let me share 3 tips based upon execution failures to improve your test planning.

I have selected the tips based upon 3 layers of what I’m calling Agile test planning. Such that as a Tester, you are required to do “Release Planning” or asked to do an “Iteration Planning” or asked to do an “Individual Tester Planning”.

Let’s take an example of “Release Planning” first.

Many years ago my Employer decided to start offering solutions in the Mobile world. Before that we were only in the Desktop world. Our first solution was for Android and then we worked on a Platform that can produce Apps in Android, iOS or Windows. A year or so after this has happened, I was asked to do some test planning for a release meant for a Client in China. I ‘assumed’ that they’ll need mobile solution as it was our top priority then. So my plan started with Android and then iOS testing on Simulators and Devices.

The plan looked great. Our first iteration started and by that time, I had no interaction with the Client. Client started reporting a lot of bugs and all of them were reported in Windows Desktop environment. I learned that this Client is only interested in good old Windows Desktop solution and doesn’t care about mobile.

The lesson which I learned is my tip # 1 for this blog:

Always ask question rather than assuming things

There is an excellent resource on Ministry of Testing that gives you a good list of what questions to ask for test planning. Michael Bolton also has a list of questions.

Let’s now move to “Iteration Planning”.

In a project that I was part as the lead tester, the Product Owner complained that the number of new bugs reported towards the end of Iteration is far greater than the ones reported at the start. My first response was that as more things get ready for testing towards the end, more testing is done and hence more defects were found.

Then I gave it a deep thought by looking at the history of tests we were executed. One pattern was there that certain test plans produced higher number of defects (following the heuristic that Software Bugs are like Real bugs) and the test execution order was left on the tester to pick. Tester knew that this test plan will consume lot of time and energy, so they left these to the end.

The rule that we made afterwards is my tip # 2 for this blog:

Prioritize test execution based upon which tests will fail first

This article can help you get started on Risk Based Testing.

Let’s look at the last one called “Individual tester planning”.

When I was young and raw, I had very ambitious plans for myself assuming that I’ll get all things in my favor. I planned it 8 hours a day with no distractions. Eventually I couldn’t always finish the stuff that I committed at the start of Iteration.

With age and lot of experiences, I have become more realistic about my estimates and always consider it as a 6 hours day. I also account for unknown distractions like network outage, random meeting popping up, sick days and what not. It’s not that I give a pessimistic by including all those factors but I do keep margin for some of them.

The lesson that I learned is my tip # 3 for this blog:

A plan that doesn’t incorporate change is not a plan

What tips you can offer from your execution failures to improve test planning?

Test Execution in focus

Test execution is perhaps the most neglected area in the field of Software Testing. Too much focus is put on the test planning and design part as compared to test execution and reporting. One reason for this is that test planning/design is considered to be the intellectual part of the equation where the skills of a Tester make the most difference.

Let me try to take this topic today.

The first and foremost objective of test management for the execution phase is to ensure that all tests that have been written are executed on regular basis. The initial investment of writing Unit tests or any automated tests is paid off when those test execute for may be 1,000 times. You should always remember:

A single test that is executed thousand times is better than one thousand tests that are never executed.

You may be saying that how come a test is written but it is not executed. But it’s true. Tests get hidden in the system. At times, they are put in the ignored list to have your build passing, and no one looks at them again. At times, they are executed on only one configuration and other important configurations go missing due to either lack of resources or due to poor planning.

The next thing to consider is how much time it takes to execute them. Because if you leave things as it is on the test execution side, your testing starts becoming expensive. The old saying “time is money” is coming ever true with hardware and software being acquired as a service and paid on per hour basis. The more time you spend on test execution, the expensive it becomes.

If your test execution is all manual, it is by any standard the most expensive part of the system. And if some of it is automated, keep an eye on how much time it take to run. It may be fast but it has to be super fast.

For example in one of the projects that I’m working, we have over 5,000 unit tests that are executed with each Developer build. The overall execution time reached to 25 minutes which was even more than the actual build time. We parallelized it and time got reduced to 15 minutes. Now we are working at optimizing test startup time and data creation to even reduce the time. The saving is huge as it cuts the time for each Developer and enhances the overall Developer productivity.

Next thing to consider is how frequent your test run. If you run your Performance tests once in a week, then even if you notice a dip in performance it is really hard to pin point the actual commit message that caused the Performance to go down. If you can move it to daily, your chances suddenly are better to trace back such issues. Luckily many automation server tools help you orchestrate your jobs and you should view tools like Jenkins not only as Continuous Integration tools but as Continuous Testing tools.

The final comment on test execution is how the results are being reported. Most test execution results are raw and are in the shape of data. One thing that we did recently is to write series of Python scripts to manipulate that data and transform that into information. Such that results of our test execution are reported immediately on a Testing Dashboard in a Management friendly way like testing heat analysis . The end result is that at any time a Project Manager (or Product Manager or whoever is responsible for release decisions) can go and see what is the risk if we release just now.

How much emphasis you give to test execution phase? Have you tried something that is worth sharing with others?

Tester in the valley

Summers offer exploration options. And exploration is always good for Testers.

Couple of weeks ago, I spent few days in the picturesque valley of Neelum in Azad Jammu Kashmir. The main attraction was to enjoy the beautiful scenes and flowing waters across the way. But being Tester for life, I had a second mission: find some bugs.

We did not have to wait long.

At the entry point where Tourist information was recorded as the road touches the Line of Control (a loose border between Pakistan and India for the disputed Kashmir region), we were given a brochure from the Tourist department. It was very helpful as it had distances, important phone numbers and places of interest. But my 12 yo daughter (who has accompanied me on previous bug hunting summer excursions) spotted an issue.

Sorry for this being in Urdu but it supposed to mean the “Meeting point of Neelum and Jehlum rivers”. The funny part was that by taking the ‘mmm’ sound off the second river, it meant “Meeting point of Neelum reiver and Stupidity”. Yes we stopped there to make that meet happen.

On our way beyond this meeting point was a long travel along the Neelum river. There is a lesson of Context in the name of the river. The same river is called “Kishan Ganga” by our Indian neighbors. The funny part is that at certain points, the River itself is the border. For example in the above picture, I’d be calling the river as “Neelum” and people across the river would be calling it “Kishan Ganga”. We can have a long discussion on what is the “correct” name of the river but reality is that it is one river with two names. How familiar is this situation when we have a word fight on certain terminology and that’s why understanding each other’s point of view matters a lot. For example in building a good relationship between Developers and Testers.

English being 3rd language in our region with every valley/region having it’s own language and then Urdu as national language, there are so many spelling mistakes to be caught in the journey. For example a hotel had numbered it’s multi-story building as following:

As you can imagine that with so much greenery out there, the possibility of finding the real bugs was very high. Some of them were unique in their textures, some were disguised and some were shouting out loud to be caught:

And yes your guess is right. Where there are bugs, there are bug hunters. Some of them were also ready for a photo-shoot:

There were many other spelling mistake bugs which I caught but I’m leaving them as they were level 0 or low severity bugs 😊. But there was one thing for which I couldn’t find a bug: the beautiful landscape that had all in one picture like the one below from Arrang Kel:

And oh yeah, before I go I’d like to mention a life philosophy lesson that I found on one of Rickshaws in Muzaffarbad which read:

Reading “Saans hay to Chance hay” meaning that if you are breathing, you have a chance to do something. We get bogged down by stressing office environment, project fatigues, overwork but remember how worthy it is still being alive. Because

Saans hay to Chance hay

Have you explored any new area in your Summers? What did you find worthy of sharing?

Automation cannot be Generalized

Consider the automatic machines you have at home as an example. You have a Washing machine to do a laundry on any day that you want. And then you have Dish washing machine that can clean your crockery at your will.

Aha… someone would say, here is some common stuff. You see both of them are machines that do “washing”, so why not we generalize it and make it one machine that will be able to “wash” either clothes or plates. They both use water and some sort of “detergent” to cleanse the items we put in it; they both use rinsing as a way to take off all the dirt etc.; and they both use air to dry the items once the washing is done.

If someone approaches you with the above plan, you should immediately recognize that you are being sold a “Phakki”. This can be an excellent plot of any piece of fiction from cartoons to movies but it has nothing to do with the real world. So if someone sells you a Generalized Test Automation solution, your response should be similar to the above plan.

(Charlie Chaplin in “Modern Times” movie. Full clip is here)

The common stuff in the case of washing machine can be components that you have in both machines, like an electric motor, water drainage system, air blowers and other similar pieces. But the function that each machine performs stays unique and thus requires a unique design and implementation.

So when you approach automation of your testing activities, you can have some common components that you use in different solutions, but each solution will have it’s unique needs. For example, over last couple of years we have written many scripts to automate the following testing activities:

  • Smoke Testing
  • Performance Testing
  • Data Conversion Testing
  • Compatibility Testing
  • Cross Platform Testing
  • Memory Leak Testing

All these scripts are specialized as they target one type of testing at a time. The common pieces are just the sorts of “Get the build to test”, “Gather the results” and “Notify about results”. The actual code that automates Performance Testing is a different beast than the one we have for Compatibility Testing.

We do have a lean type of Test Automation Framework. For other cases, there can be a framework that takes a General test automation tool as a starting point and delivers a specific set of specialized implementation for your product. In one of the talks at our regular meetups, my tester friend Adeel Shoukat gave an automation tip that was roughly “You’ll need to write lot of your own test routines using tools as you cannot use the tools right off the shelf to test your application”.

Remember:

If someone tells you that here is a Generic Test Automation suite that will solve all your testing problems, you are being goofed.

Stay away from those pocket pickers.

How has been your journey towards automation? Have you ever seen or implemented a Generalized automation solution by any chance?

Testing is shifting Left

With the DevOps movement and the notion of packaging your Software as a Service (SaaS), many traditional concepts are being shaken up. Whether you like it or not but the truth is that in today’s ever changing and globalized market, the old ways to develop software are not working.

Look at facebook for example. Have you ever seen a notice like this?

“Services at facebook will remain closed from this date to that date (or for these hours) for maintenance purpose. You’ll not be able to access your accounts during this time. We regret the inconvenience caused by this.”

Obviously not because that is so much 1990’s way of upgrading an Internet based service. Facebook instead follows the mechanism of “changing the wheels while car is moving” rather I’d say “changing the engine while the jet is flying”. Yes, facebook has a policy of deploying new changes daily since long and you don’t even know when it was updated with newer features. All this has only been possible through lots of changes in the way software is developed, deployed and maintained.

One of the key features in such situations is to shift Testing to the Left. There are other significant pieces too which I’ll try to talk in some future posts but let’s now focus on Shift Left of Testing.

These pages will help you get the concept in a much better way and I’d recommend reading them. If you are one who like to get someone read articles for you and give you a nice summary, I’m here to help you with this: “Traditionally Testing is done after a feature is developed. Shift Left refers to shifting testing to left (think of classical waterfall model where things moved from left to right in phases), such that it becomes an integral part of Developing the feature”.

(The image is taken from here.)

This concept is not now. Agile Testing introduced the principle of Testing being done by all team members with Power of Three or Three Amigos. Lot of talk has been on testing early in the process for long. But in all such suggestions, the onus of testing remains on the Tester in the team and others come in as helpers. Like when Testing team says that we need to Test early (kind of let’s shift testing to the left), what they usually mean is that a Tester should become part of the design discussions and should start writing/executing tests as early as possible. But here

We want Testing to happen early regardless of who does it.

It’s kind of funny that for years we were told that Developers (though I prefer the term Programmer but writing it Developer to refer to that point of view) are superior than Testers and Testing itself is a B grade job that should go to such workers. Now all that inferior work is suddenly becoming such an important task that it will be done by the Developer.

What does it mean for you if you are a Programmer?

Yes, you got it right. As a Programmer, you should be writing tests as you write the feature. As Matt Brigs states in this post:

A senior developer will understand that this job is to provide solutions to problems, not write code.

So become a Senior Developer all of sudden by realizing this. Write tests and run them even before you commit changes to ensure that solution is being written, not the code.

What does it mean for you if you are a Tester?

You need to let go of the mentality that “Testing is something that Testers do”. Rather you should help the team do testing by improving the Testing machinery or the tooling needed by the team. Spend more time in helping Programmers write more tests. Make systems that help run Continuous Testing as part of each step as Dan Ashby mentions that we test even more in DevOps. Remember that Rob Lambert  shared a lesson during his company’s transition to  weekly releases from yearly releases:

Testing as an activity, has to become central to the team.

In our team, we have started this shift recently and I hope to write more on the lessons learned in coming weeks and months. Wish us luck!

Have your team joined Shift Lift movement yet? What are your observations so far?

Using Logs for Performance Testing

Performance testing is one of the specialty area for testers and is one of the challenging tasks. Designing Performance tests is different than designing functional tests, then setting up testing environment for Performance also needs special care and finally figuring out if the results do indicate an issue is not straight forward. One thing that has helped us lately in troubleshooting Performance issue is logging.

It is a known fact that logging is a Tester’s friend. Rather I’d say that it is a friend for any troubleshooter. The logs are like the traces which a skilled investigator follows to find the culprit who committed the crime. These are “Khurras” (Punjabi meaning foot prints) which “Khojis” (Punjabi/Urdu/Hindi meaning Detectors) follow. If the crime was committed (read an issue was observed) but no “khurras” are there (read no logs are there), it is not possible for the “Khoji” to find out the criminal (read what is the root cause of the issue).

(Original photo is here)

The first question in setting up logs is what logs we need? These have to be key points in software behavior for which time/memory captures are added to the log. For example in one of our Display systems where we implemented logging, we added logging for the following:

  • The time to open the file.
  • The time to display the thumbnails in the file.
  • The time to fill view with selected thumbnail.
  • The time to show properties when an element is selected in the display.

Once you know what points you need to log, the next step is to hook in logging. I’m hoping that your software or the tools/technologies that is being used by your software already has logging infrastructure in it and you have to add a new logger for Performance. And if it is unfortunately not there, then you need to have such infrastructure first. This logging will be added by the Programmers in your team and thus this project is another example of how Programmers and Testers can work together to deliver quality.

Suppose you have those friendly Programmers in your team and Performance logging is enabled. Now all you have to do is turn it on and run your existing tests. That’s it. Along with the numbers that measured before, you’ll get the logs.

Remember when all is going good, no one needs logs.

So you’ll need the logs only when your results are indicated a decline in Performance numbers that you measure. And then you can pass it on logs directly to the Programmers to investigate. Or if you are a smart tester like me, you’d write a small Python script to parse the log and get the meaningful numbers that you report to your team.

We’ve tried the above and believe it is really helping in issue investigation. We have moved through this from “We have poor Performance numbers” to “The Performance is down because file goes through up gradation progress while opening” like concrete issues being reported and fixed. Try it and you’ll love it. It will take some effort from both the Programmers and Testers to do it but it is surely worth an effort if you are serious about Performance testing.

Do you use logging in Performance testing? Or you have some other interesting ways to investigate Performance issues?

3 Misconceptions about Testing Career

Last week I was invited to speak to graduating batch of BS Computer Science at FAST Islamabad Campus. This was part of an initiative through which Industry professionals visit students and give them a true picture of real world. On my previous visits, I have talked about different aspects of Software Testing whereas this time I picked up misconceptions.

There is a long list of misconceptions but here were the 3 that I talked about:

  1. Don’t like Coding; Become a Tester.

That’s number one as many in Academia still believe Testing to be a non-coding job. That may have been true in 1990s when the profession of Software Testing was new and the gap was filled by people from other professions who were expert users and became testers. But the latest job market requires more and more testers who can write code to test code.

I shared examples of my employer’s recent hiring for Automation analyst and also shared SDET positions being announced in local market.

  1. Testers are paid less than Programmers.

This may have come in due to earlier practice of Industry to pay less to Testers who were Clickers in some respect. The pay scale of job reflects its complexity and it was fair to pay dumb testers less than Programmers. But again as the recent demand of smart and knowledgeable testers who are in some ways Gray Box Testers, the pay scale need to match that of a Programmer as both roles require intellectual work.

I used the survey data from P@SHA to share Entry level programmer salary vs. that of a Tester which suggests both are in the same league.

  1. Testers don’t grow. All important positions go to Programmers.

With testing becoming central to modern day Apps, testers are growing in multiple directions. There are testers who grow to run the entire testing department of the organizations. There are testers who grow to lead Engineering departments. And there are testers who become Testing experts in Performance testing or Automation or some other domain.

I used the Speakers list from our PSQC’17 that was held last month to give examples from each category. That really helped because local examples work more than any thing and I realized that holding a Software Quality Conference is helpful in many ways.

In the end there were many questions around the above myths and the realities involved. One question was aimed towards bitter relationship between Testers and Programmers for which I answered that this behavior needs to go and we need friendship between them to be able to deliver Quality Software.

What else you would like to add to this list of misconceptions?