Test Execution in focus

Test execution is perhaps the most neglected area in the field of Software Testing. Too much focus is put on the test planning and design part as compared to test execution and reporting. One reason for this is that test planning/design is considered to be the intellectual part of the equation where the skills of a Tester make the most difference.

Let me try to take this topic today.

The first and foremost objective of test management for the execution phase is to ensure that all tests that have been written are executed on regular basis. The initial investment of writing Unit tests or any automated tests is paid off when those test execute for may be 1,000 times. You should always remember:

A single test that is executed thousand times is better than one thousand tests that are never executed.

You may be saying that how come a test is written but it is not executed. But it’s true. Tests get hidden in the system. At times, they are put in the ignored list to have your build passing, and no one looks at them again. At times, they are executed on only one configuration and other important configurations go missing due to either lack of resources or due to poor planning.

The next thing to consider is how much time it takes to execute them. Because if you leave things as it is on the test execution side, your testing starts becoming expensive. The old saying “time is money” is coming ever true with hardware and software being acquired as a service and paid on per hour basis. The more time you spend on test execution, the expensive it becomes.

If your test execution is all manual, it is by any standard the most expensive part of the system. And if some of it is automated, keep an eye on how much time it take to run. It may be fast but it has to be super fast.

For example in one of the projects that I’m working, we have over 5,000 unit tests that are executed with each Developer build. The overall execution time reached to 25 minutes which was even more than the actual build time. We parallelized it and time got reduced to 15 minutes. Now we are working at optimizing test startup time and data creation to even reduce the time. The saving is huge as it cuts the time for each Developer and enhances the overall Developer productivity.

Next thing to consider is how frequent your test run. If you run your Performance tests once in a week, then even if you notice a dip in performance it is really hard to pin point the actual commit message that caused the Performance to go down. If you can move it to daily, your chances suddenly are better to trace back such issues. Luckily many automation server tools help you orchestrate your jobs and you should view tools like Jenkins not only as Continuous Integration tools but as Continuous Testing tools.

The final comment on test execution is how the results are being reported. Most test execution results are raw and are in the shape of data. One thing that we did recently is to write series of Python scripts to manipulate that data and transform that into information. Such that results of our test execution are reported immediately on a Testing Dashboard in a Management friendly way like testing heat analysis . The end result is that at any time a Project Manager (or Product Manager or whoever is responsible for release decisions) can go and see what is the risk if we release just now.

How much emphasis you give to test execution phase? Have you tried something that is worth sharing with others?

Tester in the valley

Summers offer exploration options. And exploration is always good for Testers.

Couple of weeks ago, I spent few days in the picturesque valley of Neelum in Azad Jammu Kashmir. The main attraction was to enjoy the beautiful scenes and flowing waters across the way. But being Tester for life, I had a second mission: find some bugs.

We did not have to wait long.

At the entry point where Tourist information was recorded as the road touches the Line of Control (a loose border between Pakistan and India for the disputed Kashmir region), we were given a brochure from the Tourist department. It was very helpful as it had distances, important phone numbers and places of interest. But my 12 yo daughter (who has accompanied me on previous bug hunting summer excursions) spotted an issue.

Sorry for this being in Urdu but it supposed to mean the “Meeting point of Neelum and Jehlum rivers”. The funny part was that by taking the ‘mmm’ sound off the second river, it meant “Meeting point of Neelum reiver and Stupidity”. Yes we stopped there to make that meet happen.

On our way beyond this meeting point was a long travel along the Neelum river. There is a lesson of Context in the name of the river. The same river is called “Kishan Ganga” by our Indian neighbors. The funny part is that at certain points, the River itself is the border. For example in the above picture, I’d be calling the river as “Neelum” and people across the river would be calling it “Kishan Ganga”. We can have a long discussion on what is the “correct” name of the river but reality is that it is one river with two names. How familiar is this situation when we have a word fight on certain terminology and that’s why understanding each other’s point of view matters a lot. For example in building a good relationship between Developers and Testers.

English being 3rd language in our region with every valley/region having it’s own language and then Urdu as national language, there are so many spelling mistakes to be caught in the journey. For example a hotel had numbered it’s multi-story building as following:

As you can imagine that with so much greenery out there, the possibility of finding the real bugs was very high. Some of them were unique in their textures, some were disguised and some were shouting out loud to be caught:

And yes your guess is right. Where there are bugs, there are bug hunters. Some of them were also ready for a photo-shoot:

There were many other spelling mistake bugs which I caught but I’m leaving them as they were level 0 or low severity bugs 😊. But there was one thing for which I couldn’t find a bug: the beautiful landscape that had all in one picture like the one below from Arrang Kel:

And oh yeah, before I go I’d like to mention a life philosophy lesson that I found on one of Rickshaws in Muzaffarbad which read:

Reading “Saans hay to Chance hay” meaning that if you are breathing, you have a chance to do something. We get bogged down by stressing office environment, project fatigues, overwork but remember how worthy it is still being alive. Because

Saans hay to Chance hay

Have you explored any new area in your Summers? What did you find worthy of sharing?

Automation cannot be Generalized

Consider the automatic machines you have at home as an example. You have a Washing machine to do a laundry on any day that you want. And then you have Dish washing machine that can clean your crockery at your will.

Aha… someone would say, here is some common stuff. You see both of them are machines that do “washing”, so why not we generalize it and make it one machine that will be able to “wash” either clothes or plates. They both use water and some sort of “detergent” to cleanse the items we put in it; they both use rinsing as a way to take off all the dirt etc.; and they both use air to dry the items once the washing is done.

If someone approaches you with the above plan, you should immediately recognize that you are being sold a “Phakki”. This can be an excellent plot of any piece of fiction from cartoons to movies but it has nothing to do with the real world. So if someone sells you a Generalized Test Automation solution, your response should be similar to the above plan.

(Charlie Chaplin in “Modern Times” movie. Full clip is here)

The common stuff in the case of washing machine can be components that you have in both machines, like an electric motor, water drainage system, air blowers and other similar pieces. But the function that each machine performs stays unique and thus requires a unique design and implementation.

So when you approach automation of your testing activities, you can have some common components that you use in different solutions, but each solution will have it’s unique needs. For example, over last couple of years we have written many scripts to automate the following testing activities:

  • Smoke Testing
  • Performance Testing
  • Data Conversion Testing
  • Compatibility Testing
  • Cross Platform Testing
  • Memory Leak Testing

All these scripts are specialized as they target one type of testing at a time. The common pieces are just the sorts of “Get the build to test”, “Gather the results” and “Notify about results”. The actual code that automates Performance Testing is a different beast than the one we have for Compatibility Testing.

We do have a lean type of Test Automation Framework. For other cases, there can be a framework that takes a General test automation tool as a starting point and delivers a specific set of specialized implementation for your product. In one of the talks at our regular meetups, my tester friend Adeel Shoukat gave an automation tip that was roughly “You’ll need to write lot of your own test routines using tools as you cannot use the tools right off the shelf to test your application”.


If someone tells you that here is a Generic Test Automation suite that will solve all your testing problems, you are being goofed.

Stay away from those pocket pickers.

How has been your journey towards automation? Have you ever seen or implemented a Generalized automation solution by any chance?

Testing is shifting Left

With the DevOps movement and the notion of packaging your Software as a Service (SaaS), many traditional concepts are being shaken up. Whether you like it or not but the truth is that in today’s ever changing and globalized market, the old ways to develop software are not working.

Look at facebook for example. Have you ever seen a notice like this?

“Services at facebook will remain closed from this date to that date (or for these hours) for maintenance purpose. You’ll not be able to access your accounts during this time. We regret the inconvenience caused by this.”

Obviously not because that is so much 1990’s way of upgrading an Internet based service. Facebook instead follows the mechanism of “changing the wheels while car is moving” rather I’d say “changing the engine while the jet is flying”. Yes, facebook has a policy of deploying new changes daily since long and you don’t even know when it was updated with newer features. All this has only been possible through lots of changes in the way software is developed, deployed and maintained.

One of the key features in such situations is to shift Testing to the Left. There are other significant pieces too which I’ll try to talk in some future posts but let’s now focus on Shift Left of Testing.

These pages will help you get the concept in a much better way and I’d recommend reading them. If you are one who like to get someone read articles for you and give you a nice summary, I’m here to help you with this: “Traditionally Testing is done after a feature is developed. Shift Left refers to shifting testing to left (think of classical waterfall model where things moved from left to right in phases), such that it becomes an integral part of Developing the feature”.

(The image is taken from here.)

This concept is not now. Agile Testing introduced the principle of Testing being done by all team members with Power of Three or Three Amigos. Lot of talk has been on testing early in the process for long. But in all such suggestions, the onus of testing remains on the Tester in the team and others come in as helpers. Like when Testing team says that we need to Test early (kind of let’s shift testing to the left), what they usually mean is that a Tester should become part of the design discussions and should start writing/executing tests as early as possible. But here

We want Testing to happen early regardless of who does it.

It’s kind of funny that for years we were told that Developers (though I prefer the term Programmer but writing it Developer to refer to that point of view) are superior than Testers and Testing itself is a B grade job that should go to such workers. Now all that inferior work is suddenly becoming such an important task that it will be done by the Developer.

What does it mean for you if you are a Programmer?

Yes, you got it right. As a Programmer, you should be writing tests as you write the feature. As Matt Brigs states in this post:

A senior developer will understand that this job is to provide solutions to problems, not write code.

So become a Senior Developer all of sudden by realizing this. Write tests and run them even before you commit changes to ensure that solution is being written, not the code.

What does it mean for you if you are a Tester?

You need to let go of the mentality that “Testing is something that Testers do”. Rather you should help the team do testing by improving the Testing machinery or the tooling needed by the team. Spend more time in helping Programmers write more tests. Make systems that help run Continuous Testing as part of each step as Dan Ashby mentions that we test even more in DevOps. Remember that Rob Lambert  shared a lesson during his company’s transition to  weekly releases from yearly releases:

Testing as an activity, has to become central to the team.

In our team, we have started this shift recently and I hope to write more on the lessons learned in coming weeks and months. Wish us luck!

Have your team joined Shift Lift movement yet? What are your observations so far?

Using Logs for Performance Testing

Performance testing is one of the specialty area for testers and is one of the challenging tasks. Designing Performance tests is different than designing functional tests, then setting up testing environment for Performance also needs special care and finally figuring out if the results do indicate an issue is not straight forward. One thing that has helped us lately in troubleshooting Performance issue is logging.

It is a known fact that logging is a Tester’s friend. Rather I’d say that it is a friend for any troubleshooter. The logs are like the traces which a skilled investigator follows to find the culprit who committed the crime. These are “Khurras” (Punjabi meaning foot prints) which “Khojis” (Punjabi/Urdu/Hindi meaning Detectors) follow. If the crime was committed (read an issue was observed) but no “khurras” are there (read no logs are there), it is not possible for the “Khoji” to find out the criminal (read what is the root cause of the issue).

(Original photo is here)

The first question in setting up logs is what logs we need? These have to be key points in software behavior for which time/memory captures are added to the log. For example in one of our Display systems where we implemented logging, we added logging for the following:

  • The time to open the file.
  • The time to display the thumbnails in the file.
  • The time to fill view with selected thumbnail.
  • The time to show properties when an element is selected in the display.

Once you know what points you need to log, the next step is to hook in logging. I’m hoping that your software or the tools/technologies that is being used by your software already has logging infrastructure in it and you have to add a new logger for Performance. And if it is unfortunately not there, then you need to have such infrastructure first. This logging will be added by the Programmers in your team and thus this project is another example of how Programmers and Testers can work together to deliver quality.

Suppose you have those friendly Programmers in your team and Performance logging is enabled. Now all you have to do is turn it on and run your existing tests. That’s it. Along with the numbers that measured before, you’ll get the logs.

Remember when all is going good, no one needs logs.

So you’ll need the logs only when your results are indicated a decline in Performance numbers that you measure. And then you can pass it on logs directly to the Programmers to investigate. Or if you are a smart tester like me, you’d write a small Python script to parse the log and get the meaningful numbers that you report to your team.

We’ve tried the above and believe it is really helping in issue investigation. We have moved through this from “We have poor Performance numbers” to “The Performance is down because file goes through up gradation progress while opening” like concrete issues being reported and fixed. Try it and you’ll love it. It will take some effort from both the Programmers and Testers to do it but it is surely worth an effort if you are serious about Performance testing.

Do you use logging in Performance testing? Or you have some other interesting ways to investigate Performance issues?

3 Misconceptions about Testing Career

Last week I was invited to speak to graduating batch of BS Computer Science at FAST Islamabad Campus. This was part of an initiative through which Industry professionals visit students and give them a true picture of real world. On my previous visits, I have talked about different aspects of Software Testing whereas this time I picked up misconceptions.

There is a long list of misconceptions but here were the 3 that I talked about:

  1. Don’t like Coding; Become a Tester.

That’s number one as many in Academia still believe Testing to be a non-coding job. That may have been true in 1990s when the profession of Software Testing was new and the gap was filled by people from other professions who were expert users and became testers. But the latest job market requires more and more testers who can write code to test code.

I shared examples of my employer’s recent hiring for Automation analyst and also shared SDET positions being announced in local market.

  1. Testers are paid less than Programmers.

This may have come in due to earlier practice of Industry to pay less to Testers who were Clickers in some respect. The pay scale of job reflects its complexity and it was fair to pay dumb testers less than Programmers. But again as the recent demand of smart and knowledgeable testers who are in some ways Gray Box Testers, the pay scale need to match that of a Programmer as both roles require intellectual work.

I used the survey data from P@SHA to share Entry level programmer salary vs. that of a Tester which suggests both are in the same league.

  1. Testers don’t grow. All important positions go to Programmers.

With testing becoming central to modern day Apps, testers are growing in multiple directions. There are testers who grow to run the entire testing department of the organizations. There are testers who grow to lead Engineering departments. And there are testers who become Testing experts in Performance testing or Automation or some other domain.

I used the Speakers list from our PSQC’17 that was held last month to give examples from each category. That really helped because local examples work more than any thing and I realized that holding a Software Quality Conference is helpful in many ways.

In the end there were many questions around the above myths and the realities involved. One question was aimed towards bitter relationship between Testers and Programmers for which I answered that this behavior needs to go and we need friendship between them to be able to deliver Quality Software.

What else you would like to add to this list of misconceptions?

Pakistan Software Quality Conference 2017

The first ever conference about Software Quality and perhaps the biggest Software Engineering conference in Pakistan’s history happened last weekend. Called as PSQC17 and organized through the platform of PSTB (Pakistan Software Testing Board), it drew ~275 Software Quality practitioners and researchers across the country.

This was a dream come true for all of us who have been building the community over the last three years through various events. And what a rewarding manifestation of our dream it was on the day. The registration desk opened at 8:30 AM and some testers were there already. Within next half an hour of Registration & Networking time, the beautiful Crystal Ball Room at the Marriott, Islamabad were filling rapidly. The proceedings started with recitation of Holy Quran and then the host of the event Roshan Masood from Confiz Ltd, Lahore took over the stage. With a smiling welcome to all and explaining the Conference theme “Testing in the Real”, Roshan invited President PSTB Dr. Zohaib Iqbal to give the opening note.

Dr. Zohaib first explained why talking about Software Quality is so important in today’s complex world which runs on software. He introduced the PSTB and it’s different objectives that include building a community and sharing knowledge. He briefly recapped the journey so far and how this Conference is a perfect climax to all those activities. Dr. Zohaib also shared demographics of the audience which included participation from all the provinces, almost one third audience representing female population and a mixed audience with varying experience. He set up some expectations for the enthusiastic gathering in the hall and laid down structure of the talks coming their way.

(more photos coming soon to facebook event page)

On this occasion, we were honored by the presence of Ambassadors in Pakistan from Portugal, Tunisia, Ukraine, Bulgaria and guests from Japan and China Embassy. Taking advantage to get in touch with top IT guys of the country, Portuguese Ambassador Joao Paulo Sabido Costa Addressed and shared how IT industry from both countries can work together for collective success. The Tunisian Ambassador Adel Elarbi also welcomed all the participants and extended his support in building the bridges between skillful people from both countries.

Here the first session of making connections finished with some refreshments offered to everyone. The testers from Lahore and Karachi were warmly welcomed by the ones from twin cities of Islamabad and Rawalpindi. I saw many new connections being made there which I can assure will go a long way.

The proceedings resumed through a talk on Quality Today by Kamran Shaukat Ali Khan, AVP Systems Ltd. who traveled from Lahore for the event. Kamran’s talk revolved around the basic questions of what Quality is, how we define it, what it costs to produce Quality and why we still see so many failures. Through references to classic materials like Quality is Free by Crosby and Deming’s principles, he involved the audience by asking many questions. He summed up by motivating the Testers to step up and take overall ownership of Quality in the products and solutions they deliver.  You can get the whole presentation here: PSQC17 – Quality Today – Kamran Shaukat

After this talk, we had to split the session in parallel. Many were not happy as they wanted to attend all the sessions and they were reminded that “Life is parallel and so is learning”.

At this point sessions were held back to back both in the main Conference Hall and the Ambassador Room. I moved around all the places to catch maximum on talks but some of my notes are based upon observations from others who attended the sessions.

In the main hall, next talk was on 5 Cs of DevOps by Syed Azhar ul Islam, Director DevOps Careem who flew from Karachi specially for the event. Azhar explained that to support products like Careem, the silos of Development, Testing, Operations, Product Management etc. need to be broken and all should work together. The summary slide had Planning drive Integration which drives Testing which gears Deployment which drives Monitoring & Feedback. All his slides are here: PSQC17 – 5Cs of DevOps – Syed Azhar ul Islam

We were being honored by so many guests in the session and I’ll just name a few Ather Imran (CEO Sybrid and President Open Islamabad), Kashif Moeen (CTO, Zigron), Dr. Amir Mahmood (Rector NU-FAST) and Dr. Mukhtar Ahmed from HEC.

In the other room, Syed Muhammad Afif Zaidi, Director Technology Pakistan S&P Global shared his journey of Building a QA setup from scratch as he has built a whole testing empire which he started as a lone tester. He gave many tips for the hiring process, building the teams, nurturing the talent, setting higher standards and constantly asking for more from the team.

Back in the main hall, next it was me talking about Testing without Specs. I started the talk by first sharing my personal experience so as how changing requirements which are outside our control make the Specification look like Specifiction (a term borrowed from Helena’s tweet). I then gave two tips to curb around it which were: talking to your team members to understand their mental maps and spending more time understanding the internals of the system you test. My last quote that “Devil is in the details, so is testing” got quite popular on the day. All the slides are here: PSQC17 – Testing Without Specs – Majd Uddin

This is the talk that I surely missed as it was parallel to mine though I so much wanted to sit there. Talking about Cost Effective Test Automation using Cloud  Adeel Shoukat (SQA Analyst) and Ehsen Raza (Senior SQA Analysst) from Contour Software Lahore, first shared the typical problem that each test automation project faces. Their experience along this journey came handy for the audience who could relate to this. They also showed small clips of the amazing work that their team is doing for others to get ideas and get inspired from. Their slides are here: PSQC17 – Cost-effective Test Automation Using Cloud Solution – Adeel Shoukat and Ehsen Raza

The last talk in the main hall before lunch was Open Source tools for DevOps testing by Adnan Maqsood Test Lead, LMKT, Islamabad. Adnan after spending some time explaining how testing fits into DevOps talked about the popular tools being used including Selenium, Appium, TestNG, Jenkins & Maven through some examples. His slides are here: PSQC17 – Open Source Tools for DevOps – Adnan Maqsood

In the other room, Shaima Niaz  Senior Software Quality Automation Engineer, Tkxel, Lahore was presenting Agile Test Automation. She started off with results of a survey on the subject conducted by her for the local Industry. One of the suggestion from her was to move to “Test and Dev” from “Test vs. Dev”. She then took examples from various recent concepts that help test and deliver products at fast pace including Industrial Automation, IoT testing and the concept of OTA (Over the Air). Her complete deck is here: PSQC17 – Agile Test Automation – Shaima Niaz

I’m sure that after reading so much new stuff, you need a break now. So you can imagine how the participants of PSQC17 were feeling after being filled with loads of information. Thus we decided to fill in the stomachs instead and use that time as an excuse to ask follow up questions from the Speakers. After that we resumed the session. Did I mention that Chicken Tikka Boti was very delicious.

Many media representative were attending the Confernece and they took these breaks as a good way to talk to the Speakrs and audience. PSQC17 were reported on at least 5 TV channels and you can watch one such report here.

It was some tutorial time after the break in the Ambassador Room where Amir Shahzad (Manager QA) and Hasan Farooq (Test Automation Engineer) from Stella Technologies, Islamabad presented Data driven testing using Spock Framework. Amir started the proceedings by explaining the concept of REST API testing and how data driven testing technique comes handy. He also shared his rational on why they picked Spock over other solutions. Hasan then showed some code examples of it’s implementation. There were many who liked this and were found discussing that they’ll try something similar in their own environments. The slides are here: PSQC17 – API Testing using Spock – Amir Shahzad and Hassan Farooq

In the meanwhile in the main hall, our own Dr. Uzair Khan from Quest Lab was sharing some real time implementation details of their research on Model Based Testing. He first maintained the stance that test automation is much bigger than just automating the test execution and if we can model the software behavior, we can achieve greater goals. He then shared some examples of using models to test popular video game Mario Bros and also example of using it in Embedded systems. His interesting slides are here: PSQC17 – Using Models to Automate Testing – Dr. Uzair Khan

Part of the PSQC17 agenda was to expand our collaboration with other domains and we had Nahil Mahmood come from Lahore to talk about Information Security as the last talk. Nahil who is CEO Delta Tech and is Chairman-designate of Pakistan Cyber Security Alliance, started off proceedings by sharing different reports depicting the current state of Information Security measures across the world and specially in Asia. He confirmed through multiple sources that situation in Pakistan is really bad and we need to take action on that. He then walked through al steps that can be taken for Software Security as part of SDLC what he called as sec-SDLC i.e. Secure SDLC. He summed up with “Software Quality includes Security. Let’s own it”. His slides are here: PSQC17 – Information Security in Pakistan – Nahil Mahmood

The Announcement from stage that #psqc17 is top trend of the day on the twitter made everyone happy and be proud of our social media team lead by Faiza Yousuf and supported by Saima Karim.

The afternoon nap was catching up everyone included me so we hurried towards a tea break. Also the participants gathered around the Registration desk one more time to collect their Certificates of Participation.

After this short break, we had a panel discussion on Future of Software Quality in Pakistan. The panelists were Shafiq Ahmed (Country Manager Stella Technologies Pakistan), Faiza Yousuf (Chief Consultatn at Outtabox who came from Karachi exclusively for this event) and Taimur Sarwar (Manager Operations at S&P Global Pakistan). They all shared their view of the future and then the hall was opened for questions. There were talks about how future students can be made more Quality conscious through modifications to the curriculu; there were discussion on why each of us need to step up and spend consistently on learning new technologies and techniques and there were many points to collaborate together to succeed together.

I was asked at the end to give a Closing Note in which I thanked through the core of my heart all Speakers, participants, our Sponsors. I then invited the whole organizing team (consisting of Dr. Zohaib Iqbal, Dr. Uzair Khan, Amir Shahzad, Hanan Atif, Farhana Adeel, Hina Zafar, Taimur Sarwar, Kashif Butt, Syed Qambar Ali, Zeeshan Asghar, Osman Ehsan, Haider Farooq) and all the volunteers who worked day in and day out to make this all happen. The team got a well deserved round of applause.

The final announcement was that PSQC17 is start of the journey, not the end and PSQC18 will be held in Lahore!

The feeling and emotions of everyone in the hall at the end were really great where they had a sense of belonging to this vibrant community, they were overwhelmed by such wonderful work being done in our own Industry and were committed to apply the ideas they learned in the day.

Long live Pakistan! Long live Testers from Pakistan!