Test execution is perhaps the most neglected area in the field of Software Testing. Too much focus is put on the test planning and design part as compared to test execution and reporting. One reason for this is that test planning/design is considered to be the intellectual part of the equation where the skills of a Tester make the most difference.
Let me try to take this topic today.
The first and foremost objective of test management for the execution phase is to ensure that all tests that have been written are executed on regular basis. The initial investment of writing Unit tests or any automated tests is paid off when those test execute for may be 1,000 times. You should always remember:
A single test that is executed thousand times is better than one thousand tests that are never executed.
You may be saying that how come a test is written but it is not executed. But it’s true. Tests get hidden in the system. At times, they are put in the ignored list to have your build passing, and no one looks at them again. At times, they are executed on only one configuration and other important configurations go missing due to either lack of resources or due to poor planning.
The next thing to consider is how much time it takes to execute them. Because if you leave things as it is on the test execution side, your testing starts becoming expensive. The old saying “time is money” is coming ever true with hardware and software being acquired as a service and paid on per hour basis. The more time you spend on test execution, the expensive it becomes.
If your test execution is all manual, it is by any standard the most expensive part of the system. And if some of it is automated, keep an eye on how much time it take to run. It may be fast but it has to be super fast.
For example in one of the projects that I’m working, we have over 5,000 unit tests that are executed with each Developer build. The overall execution time reached to 25 minutes which was even more than the actual build time. We parallelized it and time got reduced to 15 minutes. Now we are working at optimizing test startup time and data creation to even reduce the time. The saving is huge as it cuts the time for each Developer and enhances the overall Developer productivity.
Next thing to consider is how frequent your test run. If you run your Performance tests once in a week, then even if you notice a dip in performance it is really hard to pin point the actual commit message that caused the Performance to go down. If you can move it to daily, your chances suddenly are better to trace back such issues. Luckily many automation server tools help you orchestrate your jobs and you should view tools like Jenkins not only as Continuous Integration tools but as Continuous Testing tools.
The final comment on test execution is how the results are being reported. Most test execution results are raw and are in the shape of data. One thing that we did recently is to write series of Python scripts to manipulate that data and transform that into information. Such that results of our test execution are reported immediately on a Testing Dashboard in a Management friendly way like testing heat analysis . The end result is that at any time a Project Manager (or Product Manager or whoever is responsible for release decisions) can go and see what is the risk if we release just now.
How much emphasis you give to test execution phase? Have you tried something that is worth sharing with others?
Performance testing is one of the specialty area for testers and is one of the challenging tasks. Designing Performance tests is different than designing functional tests, then setting up testing environment for Performance also needs special care and finally figuring out if the results do indicate an issue is not straight forward. One thing that has helped us lately in troubleshooting Performance issue is logging.
It is a known fact that logging is a Tester’s friend. Rather I’d say that it is a friend for any troubleshooter. The logs are like the traces which a skilled investigator follows to find the culprit who committed the crime. These are “Khurras” (Punjabi meaning foot prints) which “Khojis” (Punjabi/Urdu/Hindi meaning Detectors) follow. If the crime was committed (read an issue was observed) but no “khurras” are there (read no logs are there), it is not possible for the “Khoji” to find out the criminal (read what is the root cause of the issue).
(Original photo is here)
The first question in setting up logs is what logs we need? These have to be key points in software behavior for which time/memory captures are added to the log. For example in one of our Display systems where we implemented logging, we added logging for the following:
- The time to open the file.
- The time to display the thumbnails in the file.
- The time to fill view with selected thumbnail.
- The time to show properties when an element is selected in the display.
Once you know what points you need to log, the next step is to hook in logging. I’m hoping that your software or the tools/technologies that is being used by your software already has logging infrastructure in it and you have to add a new logger for Performance. And if it is unfortunately not there, then you need to have such infrastructure first. This logging will be added by the Programmers in your team and thus this project is another example of how Programmers and Testers can work together to deliver quality.
Suppose you have those friendly Programmers in your team and Performance logging is enabled. Now all you have to do is turn it on and run your existing tests. That’s it. Along with the numbers that measured before, you’ll get the logs.
Remember when all is going good, no one needs logs.
So you’ll need the logs only when your results are indicated a decline in Performance numbers that you measure. And then you can pass it on logs directly to the Programmers to investigate. Or if you are a smart tester like me, you’d write a small Python script to parse the log and get the meaningful numbers that you report to your team.
We’ve tried the above and believe it is really helping in issue investigation. We have moved through this from “We have poor Performance numbers” to “The Performance is down because file goes through up gradation progress while opening” like concrete issues being reported and fixed. Try it and you’ll love it. It will take some effort from both the Programmers and Testers to do it but it is surely worth an effort if you are serious about Performance testing.
Do you use logging in Performance testing? Or you have some other interesting ways to investigate Performance issues?
Last week I was invited to speak to graduating batch of BS Computer Science at FAST Islamabad Campus. This was part of an initiative through which Industry professionals visit students and give them a true picture of real world. On my previous visits, I have talked about different aspects of Software Testing whereas this time I picked up misconceptions.
There is a long list of misconceptions but here were the 3 that I talked about:
- Don’t like Coding; Become a Tester.
That’s number one as many in Academia still believe Testing to be a non-coding job. That may have been true in 1990s when the profession of Software Testing was new and the gap was filled by people from other professions who were expert users and became testers. But the latest job market requires more and more testers who can write code to test code.
- Testers are paid less than Programmers.
This may have come in due to earlier practice of Industry to pay less to Testers who were Clickers in some respect. The pay scale of job reflects its complexity and it was fair to pay dumb testers less than Programmers. But again as the recent demand of smart and knowledgeable testers who are in some ways Gray Box Testers, the pay scale need to match that of a Programmer as both roles require intellectual work.
- Testers don’t grow. All important positions go to Programmers.
With testing becoming central to modern day Apps, testers are growing in multiple directions. There are testers who grow to run the entire testing department of the organizations. There are testers who grow to lead Engineering departments. And there are testers who become Testing experts in Performance testing or Automation or some other domain.
I used the Speakers list from our PSQC’17 that was held last month to give examples from each category. That really helped because local examples work more than any thing and I realized that holding a Software Quality Conference is helpful in many ways.
In the end there were many questions around the above myths and the realities involved. One question was aimed towards bitter relationship between Testers and Programmers for which I answered that this behavior needs to go and we need friendship between them to be able to deliver Quality Software.
What else you would like to add to this list of misconceptions?
The first ever conference about Software Quality and perhaps the biggest Software Engineering conference in Pakistan’s history happened last weekend. Called as PSQC17 and organized through the platform of PSTB (Pakistan Software Testing Board), it drew ~275 Software Quality practitioners and researchers across the country.
This was a dream come true for all of us who have been building the community over the last three years through various events. And what a rewarding manifestation of our dream it was on the day. The registration desk opened at 8:30 AM and some testers were there already. Within next half an hour of Registration & Networking time, the beautiful Crystal Ball Room at the Marriott, Islamabad were filling rapidly. The proceedings started with recitation of Holy Quran and then the host of the event Roshan Masood from Confiz Ltd, Lahore took over the stage. With a smiling welcome to all and explaining the Conference theme “Testing in the Real”, Roshan invited President PSTB Dr. Zohaib Iqbal to give the opening note.
Dr. Zohaib first explained why talking about Software Quality is so important in today’s complex world which runs on software. He introduced the PSTB and it’s different objectives that include building a community and sharing knowledge. He briefly recapped the journey so far and how this Conference is a perfect climax to all those activities. Dr. Zohaib also shared demographics of the audience which included participation from all the provinces, almost one third audience representing female population and a mixed audience with varying experience. He set up some expectations for the enthusiastic gathering in the hall and laid down structure of the talks coming their way.
(more photos coming soon to facebook event page)
On this occasion, we were honored by the presence of Ambassadors in Pakistan from Portugal, Tunisia, Ukraine, Bulgaria and guests from Japan and China Embassy. Taking advantage to get in touch with top IT guys of the country, Portuguese Ambassador Joao Paulo Sabido Costa Addressed and shared how IT industry from both countries can work together for collective success. The Tunisian Ambassador Adel Elarbi also welcomed all the participants and extended his support in building the bridges between skillful people from both countries.
Here the first session of making connections finished with some refreshments offered to everyone. The testers from Lahore and Karachi were warmly welcomed by the ones from twin cities of Islamabad and Rawalpindi. I saw many new connections being made there which I can assure will go a long way.
The proceedings resumed through a talk on Quality Today by Kamran Shaukat Ali Khan, AVP Systems Ltd. who traveled from Lahore for the event. Kamran’s talk revolved around the basic questions of what Quality is, how we define it, what it costs to produce Quality and why we still see so many failures. Through references to classic materials like Quality is Free by Crosby and Deming’s principles, he involved the audience by asking many questions. He summed up by motivating the Testers to step up and take overall ownership of Quality in the products and solutions they deliver. You can get the whole presentation here: PSQC17 – Quality Today – Kamran Shaukat
After this talk, we had to split the session in parallel. Many were not happy as they wanted to attend all the sessions and they were reminded that “Life is parallel and so is learning”.
At this point sessions were held back to back both in the main Conference Hall and the Ambassador Room. I moved around all the places to catch maximum on talks but some of my notes are based upon observations from others who attended the sessions.
In the main hall, next talk was on 5 Cs of DevOps by Syed Azhar ul Islam, Director DevOps Careem who flew from Karachi specially for the event. Azhar explained that to support products like Careem, the silos of Development, Testing, Operations, Product Management etc. need to be broken and all should work together. The summary slide had Planning drive Integration which drives Testing which gears Deployment which drives Monitoring & Feedback. All his slides are here: PSQC17 – 5Cs of DevOps – Syed Azhar ul Islam
We were being honored by so many guests in the session and I’ll just name a few Ather Imran (CEO Sybrid and President Open Islamabad), Kashif Moeen (CTO, Zigron), Dr. Amir Mahmood (Rector NU-FAST) and Dr. Mukhtar Ahmed from HEC.
In the other room, Syed Muhammad Afif Zaidi, Director Technology Pakistan S&P Global shared his journey of Building a QA setup from scratch as he has built a whole testing empire which he started as a lone tester. He gave many tips for the hiring process, building the teams, nurturing the talent, setting higher standards and constantly asking for more from the team.
Back in the main hall, next it was me talking about Testing without Specs. I started the talk by first sharing my personal experience so as how changing requirements which are outside our control make the Specification look like Specifiction (a term borrowed from Helena’s tweet). I then gave two tips to curb around it which were: talking to your team members to understand their mental maps and spending more time understanding the internals of the system you test. My last quote that “Devil is in the details, so is testing” got quite popular on the day. All the slides are here: PSQC17 – Testing Without Specs – Majd Uddin
This is the talk that I surely missed as it was parallel to mine though I so much wanted to sit there. Talking about Cost Effective Test Automation using Cloud Adeel Shoukat (SQA Analyst) and Ehsen Raza (Senior SQA Analysst) from Contour Software Lahore, first shared the typical problem that each test automation project faces. Their experience along this journey came handy for the audience who could relate to this. They also showed small clips of the amazing work that their team is doing for others to get ideas and get inspired from. Their slides are here: PSQC17 – Cost-effective Test Automation Using Cloud Solution – Adeel Shoukat and Ehsen Raza
The last talk in the main hall before lunch was Open Source tools for DevOps testing by Adnan Maqsood Test Lead, LMKT, Islamabad. Adnan after spending some time explaining how testing fits into DevOps talked about the popular tools being used including Selenium, Appium, TestNG, Jenkins & Maven through some examples. His slides are here: PSQC17 – Open Source Tools for DevOps – Adnan Maqsood
In the other room, Shaima Niaz Senior Software Quality Automation Engineer, Tkxel, Lahore was presenting Agile Test Automation. She started off with results of a survey on the subject conducted by her for the local Industry. One of the suggestion from her was to move to “Test and Dev” from “Test vs. Dev”. She then took examples from various recent concepts that help test and deliver products at fast pace including Industrial Automation, IoT testing and the concept of OTA (Over the Air). Her complete deck is here: PSQC17 – Agile Test Automation – Shaima Niaz
I’m sure that after reading so much new stuff, you need a break now. So you can imagine how the participants of PSQC17 were feeling after being filled with loads of information. Thus we decided to fill in the stomachs instead and use that time as an excuse to ask follow up questions from the Speakers. After that we resumed the session. Did I mention that Chicken Tikka Boti was very delicious.
Many media representative were attending the Confernece and they took these breaks as a good way to talk to the Speakrs and audience. PSQC17 were reported on at least 5 TV channels and you can watch one such report here.
It was some tutorial time after the break in the Ambassador Room where Amir Shahzad (Manager QA) and Hasan Farooq (Test Automation Engineer) from Stella Technologies, Islamabad presented Data driven testing using Spock Framework. Amir started the proceedings by explaining the concept of REST API testing and how data driven testing technique comes handy. He also shared his rational on why they picked Spock over other solutions. Hasan then showed some code examples of it’s implementation. There were many who liked this and were found discussing that they’ll try something similar in their own environments. The slides are here: PSQC17 – API Testing using Spock – Amir Shahzad and Hassan Farooq
In the meanwhile in the main hall, our own Dr. Uzair Khan from Quest Lab was sharing some real time implementation details of their research on Model Based Testing. He first maintained the stance that test automation is much bigger than just automating the test execution and if we can model the software behavior, we can achieve greater goals. He then shared some examples of using models to test popular video game Mario Bros and also example of using it in Embedded systems. His interesting slides are here: PSQC17 – Using Models to Automate Testing – Dr. Uzair Khan
Part of the PSQC17 agenda was to expand our collaboration with other domains and we had Nahil Mahmood come from Lahore to talk about Information Security as the last talk. Nahil who is CEO Delta Tech and is Chairman-designate of Pakistan Cyber Security Alliance, started off proceedings by sharing different reports depicting the current state of Information Security measures across the world and specially in Asia. He confirmed through multiple sources that situation in Pakistan is really bad and we need to take action on that. He then walked through al steps that can be taken for Software Security as part of SDLC what he called as sec-SDLC i.e. Secure SDLC. He summed up with “Software Quality includes Security. Let’s own it”. His slides are here: PSQC17 – Information Security in Pakistan – Nahil Mahmood
The afternoon nap was catching up everyone included me so we hurried towards a tea break. Also the participants gathered around the Registration desk one more time to collect their Certificates of Participation.
After this short break, we had a panel discussion on Future of Software Quality in Pakistan. The panelists were Shafiq Ahmed (Country Manager Stella Technologies Pakistan), Faiza Yousuf (Chief Consultatn at Outtabox who came from Karachi exclusively for this event) and Taimur Sarwar (Manager Operations at S&P Global Pakistan). They all shared their view of the future and then the hall was opened for questions. There were talks about how future students can be made more Quality conscious through modifications to the curriculu; there were discussion on why each of us need to step up and spend consistently on learning new technologies and techniques and there were many points to collaborate together to succeed together.
I was asked at the end to give a Closing Note in which I thanked through the core of my heart all Speakers, participants, our Sponsors. I then invited the whole organizing team (consisting of Dr. Zohaib Iqbal, Dr. Uzair Khan, Amir Shahzad, Hanan Atif, Farhana Adeel, Hina Zafar, Taimur Sarwar, Kashif Butt, Syed Qambar Ali, Zeeshan Asghar, Osman Ehsan, Haider Farooq) and all the volunteers who worked day in and day out to make this all happen. The team got a well deserved round of applause.
The final announcement was that PSQC17 is start of the journey, not the end and PSQC18 will be held in Lahore!
The feeling and emotions of everyone in the hall at the end were really great where they had a sense of belonging to this vibrant community, they were overwhelmed by such wonderful work being done in our own Industry and were committed to apply the ideas they learned in the day.
Long live Pakistan! Long live Testers from Pakistan!
Lahore Qalandars may have been eliminated from PSL but the Lahore Testers have definitely proceeded to the next round of Software Testing excellence by gathering for the 3rd Meetup this weekend.
The venue this time was the beautiful FAST-NUCES auditorium and event was sponsored by Systems Ltd. through the platform of Pakistan Software Testing Board. That makes a really good combination where Industry and Academia are working together towards building the community. A special thanks to Kamran Khan who made it happen.
The event started with refreshing remarks by our host Ali Gohar. And quickly we were onto the first technical talk on the topic of “IoT Testing” by Khubaib ur Rehman from CTO247. Khubaib after explaining what we mean by Internet of Things and how complex is the world, talked briefly on different types of testing that are important in this domain. He expanded couple of testing types like Compatibility in terms of hardware variation and Performance Testing being dealing with the big data that comes in through those devices. Audience got engaged in the discussion and Ali Iqbal who is part of ICT R&D fund commented on the IoT Excellence Center being setup by ICT R&D. The slides are here: internet-of-things-testing
(more photos are here)
Khurram Bhatti who is based in Sweden and took advantage of this visit to get engaged with the Pakistani Testing community, then graced the stage and talked about “The real potential of Software Tester in Agile”. Through entertaining GIFs and some witty statements, Khurram set the tone on why testing takes the center stage in Agile world. He then shared his vision of there things that a Tester must work on to be successful in agile: Communication, Business and Technical. Khurram gave a nice example on how engine oil can be assumed as a Communication mechanism such that if the oil is not there, engine will not run log. The presentation is shared here: khurrambhatti_real-potential-of-tester-in-agile-world
Systems Ltd.’s representative Nauman Farooqui who is currently serving as VP, Project Delivery and Customer Satisfaction was invited to share his thoughts on “Producing Quality Software”. Nauman took this opportunity to share that he started his career as Software Tester and encouraged the Testers in audience to step up the game. He quickly recapped the illustrious ~40 years of journey of Systems Ltd. which is proudly the first Software House of Pakistan. The thing that I liked most on his suggestions to producing Software with top notch Quality was that the Testers should absolutely learn the Business domain they are serving. He also gave some recent examples on how talking to the Clients on regular basis results in more chances of success.
The final technical talk was on “Lessons Learned in Test Automation” by Adeel Shoukat from Contour Software. In his typical relaxed yet always to the point way, Adeel debunked some of the Automation myths and made it clear that “Record and Replay will never work”. He also strongly held his view that Automation will not replace the manual testing thus thinking of Automation as a way to reduce testing force is not a way to go. He then shared some examples from his own experience that building a framework based on a hybrid approach to solve your problem is a good way to adapt. He suggested to start simple and show the business value that Automation can do and show the value to the management that Automation adds. His slides are here: leasson_learned_in_automation
Dr. Zohaib Iqbal, President PSTB was invited to share all needed information for the upcoming Pakistan Software Quality Conference being held at Marriott hotel, Islamabad on April 8th. Dr. Zohaib recapped the whole Quality journey of these meetups that happened in Islamabad, Lahore and Karachi. And this demands for a climax like the upcoming Conference. Dr. Zohaib offered a special discounted offer to testers from Lahore and explained the “Call for Speakers” and “Sponsorship model”. More details about the Conference can be found on the website.
The best part of the event then happened. Yes the yummy food with tea but No, the delicious talks over cup of tea were more interesting. Testers were seen talking to the Speakers and hosts of the event to ask the questions which couldn’t be asked during the session; Testers were seen seeking guidance on their technical and people management issues; Testers were seen discussing ideas on how we can make these events more successful. These are just a few highlights of the over one hour talks that happened and I am still feeling awesome about them.
That’s all from the Lahore meetup and we are looking forward to see all of these brilliant testers join us at the Conference!
The date was very close to the day when people show love for their loved ones. And the wind was somewhat “Basanty”. Yet the Testers from the twin cities showed love for their profession and enjoyed some high flying testing ideas instead at the Testers Meetup that happened this weekend.
The event was hosted at the beautiful auditorium of Bahria University and was sponsored by BroadPeak Technologies . Pakistan Software Testing Board (PSTB) provided the platform where 120 Testers representing over 40 organizations joined us.
(more pictures coming soon on facebook event)
After a quick introduction by the host, the first talk was delivered by Nabeel Ansar from VizTeck . Nabeel through his personal experience shared the vision of “End to End Test Automation” where automation starts from when the Requirements are written. He showed examples written in Gherkin and implemented in specflow to demonstrate the concept. The talk resulted in lot of questions being asked for this approach and also on Automation in general which were very well answered by Nabeel. The presentation is here: end-to-end-test-automation
The next talk was on an interesting topic of “Data Science and Software Testing” by Uridah Sami from RedBuffer (also known for her blog on testing QualityFrenzy ). Uridah set the scene by explaining what Data Science is, how companies like facebook or Netflix use this concept and how products like self driving cars rely on data. She then mapped the skills needed for Data Science and Software Testing and explained how these two worlds are connected. Read more here: data-sciences-and-software-testing
The third and final Technical talk was delivered by Amir Shahzad and Hasan Farooq of Stella Technology Inc. . In first half of the presentation titled “API testing using Spock Framework”, Amir briefly touched upon modern day web architecture and the importance of testing the backend RESTful APIs. He then explained why they prefer Spock over other frameworks which was mainly for it’s ability to be used in data driven testing. Hasan then demonstrated a live session of tests written in Spock which did the testing and validated the returning values. The audience took great interest in this idea. More details here: spock-presentation
Faisal Tajammul, Managing Director of BroadPK Technologies then took over the stage and expressed his thoughts on “Why Testing is important?” Faisal spoke out of his ~20 years of experience that if people chose Software Testing as a profession because they think that Programming is tough, they are wrong because in his opinion “Testing is tougher than Programming”. He engaged the audience by asking what is the single thing that can be done to deliver Quality products and then summed up this discussion by saying that if Testing is involved early in the process, it is the best way to produce Quality product.
The stage was then handed over to our own Dr. Zohaib Iqbal who is President PSTB to share more information on the exciting upcoming event: the first ever Pakistan Software Quality Conference . Dr. Zohaib in his typical relaxed yet very professional way, recapped the journey of testers meetup and expressed his gratitude for the interest shown by both the Testing practitioners and IT companies. He then talked about more specific details about the Conference which is happening on April 8th, 2017 at Marriott Hotel Islamabad and will feature Maik Nogens (known for holding Software Testing Worldcup) as Keynote Speaker along with other interesting talks. Dr. Zoahib urged the audience to get active and if they become Conference Champions by spreading the word, they can avail some additional discounts on top of the Early Bird being offered.
Certificates were then awarded to the very able BroadPeak testing team who organized this event under the leadership of Syed Qambar Rizvi. Shields were presented to the Presenters and all testers walked towards the Cafetria to enjoy very much needed cup of tea with snacks.
This was the best time of the event where we saw scenes like this: Testers meeting old friends, Testers making new friends, Testers getting to know more about other companies and their testing processes and techniques, Testers introducing their work to others who listened etc. etc. These conversations will go a long way in the careers of each Tester for sure.
We are very much looking forward to lead all this activity to the climax of Pakistan Software Quality Conference in April. There is another meetup being held in Lahore on February 25th in this build up, so will talk about that soon.
I’m sure that you do it in some form or other with may be a different name. And if you don’t do it, I think you should.
The idea is not wholely mine but this particular adaption is mine which I have been using for many years. The model looks good to be shared with other testers to use. it has some inspiration from Google’s ACC (Attributes Components Capability) model, and I have explained earlier too but to answer a specific question only.
Okay, this is what you should do:
- Divide your testing project in parts. The best count is 9 to help you draw the result in 3 x 3 shape. On what basis you can divide: if there are modules, or components, or facets, or attributes, or keywords or whatever works to help make decisions. The below example I have shows 9 core components in the Platform that we test.
- Decide on a color coding to show the “state of testing”. Testers also use happy, sad or angry faces to reflect on the fact how good is testing for a particular component, but colors help look it like a heat analysis. As if you expose this 3 x 3 cube to heat, how different parts will look after some exposure. The common color scheme is what you already know: Red for alarming situations, Yellow for cautions and Green for all good.
- You are good to go and do the analysis. You’ll need some definition on why you are marking a box as Red vs. marking it Green. Make sure you have both data and feelings to support it.
This is how it might look which uses an Orange color too:
So how this helps. In many ways:
- As appeared in my original article about three years ago, it helps you where to place the new tester. Obviously you are concerned on the Red areas which you want to make green.
- To make release decisions. How important are Red boxes for the current release? Remember, you’ll never get all Greens, so you have to make that decision on what is your acceptance criteria.
- As a measure of progress for example, we share this heat analysis to reflect on functional code coverage for the components. I know those numbers are not the only thing to look for, but you still need some metrics to guide you if you are on track.
Have you tried this or similar models before? What did you find out?