Archive | Software Testing RSS for this section

3 illusions of Software Testing

Jerry Weinberg is my favorite author on the subject of Software Testing and I have posted about lessons from his Errors book, shared my thoughts on some types of Bugging. Lately I was very happy to finish yet another book by him called “Perfect Software and other illusions about Software Testing” which was on my reading list for log due to excellent reviews.

Now having spent almost 15 years on Software Testing domain and overall 20 years in the pursuit of Software Quality in general, my first impression was that the illusions Jerry will talk about will be one of those that I’m familiar with and/or have dealt with. But Jerry surprised me with his wealth of knowledge and interesting examples from his illustrious journey as a Software professional. I learned lot of lessons and sharing the top 3 with you.

  1. Pinpointing problems is not Testing

Some activities that I always thought to be Testing’s responsibility are actually in a grey area where these have to be shared between Testers and Programmers. Pinpointing a problem, after it is found, is one of them.

In one of the early team that I worked with required that when Testing reports a bug in the Tracking System, they should mention the first major version it was introduced. For example, if a Tester finds a bug in v 3.4, Tester would check if it existed in v 2.0 and v1.0 (if the answer to 2.0 is no). That is an activity that Jerry calls Pinpointing and suggests to differentiate it from Testing and Debugging.

Even in my current project, because we (the Testers) have the code and are able to debug an issue, our Data conversion issues are reported after pinpointing and debugging. Hence a Tester’s considerable time goes into locating the problem code after finding the problem. This is included in Testing whereas it needs some separate attention.

In Jerry’ words as he summarize this as a Common Mistake and suggests a solution:

Demanding that Testers locate every fault: This is totally a Developer’s job, because developers have the needed skills. Testers generally don’t have these skills, though at times, they may have useful hints.

  1. Providing Information is hard

Jerry talks in much detail Tester’s role as information provider. He suggests to use Satir interaction Model:

-> intake -> meaning -> significance -> response ->

There are separate chapters dedicated on each of the above and I’d encourage any Tester who is serious about their job to understand this model through reading the book. Different techniques have been discussed to improve information intake, how different people will mean different things from same observation, how to know which information is significant and finally how to make the best response accordingly.

  1. Product reviews are actually testing

In fact Jerry calls it “Testing without Machinery” and suggests that technical product reviews are actually a way to provide information and thus is Testing. He also lists some “instant Reviews” which is that if you try to review and you hear excuses like below, you know that something is wrong:

  • It’s too early to review it.
  • It’s too small to bother with. What could possibly go wrong?
  • We’ll let you know as soon as we are ready.
  • We [tested][tried][demonstrated] it and it works okay.
  • ….

It’s a treasure of information on those few pages which you can get instantly once you ask for a review. This has inspired me to write more about Peer Code Reviews so you should expect an article soon.

Let me end this with Jerry’s words:

If humans were prefect thinkers, we wouldn’t need to test our work. … But we are imperfect, irrational, value-driven, diverse human beings. Therefore, we test, and we test our testing.

Software Security workshop happened in Islamabad

Saima Karim (aka Mountain Girl) covered the “Insights into Software Security Workshop” which was conceived and conducted by Amir Shahzd.

“In Pakistan, testers meet up happen rarely and if it happens, it must be appreciated. Stella Technology organized workshop named as ‘Insight into Software Security’ last Saturday i.e. Sep 23, 2017 in their Islamabad office. Main objective of workshop was to provide testers community a space to share knowledge about Software Security.

Continue reading at original post here:

Test Execution in focus

Test execution is perhaps the most neglected area in the field of Software Testing. Too much focus is put on the test planning and design part as compared to test execution and reporting. One reason for this is that test planning/design is considered to be the intellectual part of the equation where the skills of a Tester make the most difference.

Let me try to take this topic today.

The first and foremost objective of test management for the execution phase is to ensure that all tests that have been written are executed on regular basis. The initial investment of writing Unit tests or any automated tests is paid off when those test execute for may be 1,000 times. You should always remember:

A single test that is executed thousand times is better than one thousand tests that are never executed.

You may be saying that how come a test is written but it is not executed. But it’s true. Tests get hidden in the system. At times, they are put in the ignored list to have your build passing, and no one looks at them again. At times, they are executed on only one configuration and other important configurations go missing due to either lack of resources or due to poor planning.

The next thing to consider is how much time it takes to execute them. Because if you leave things as it is on the test execution side, your testing starts becoming expensive. The old saying “time is money” is coming ever true with hardware and software being acquired as a service and paid on per hour basis. The more time you spend on test execution, the expensive it becomes.

If your test execution is all manual, it is by any standard the most expensive part of the system. And if some of it is automated, keep an eye on how much time it take to run. It may be fast but it has to be super fast.

For example in one of the projects that I’m working, we have over 5,000 unit tests that are executed with each Developer build. The overall execution time reached to 25 minutes which was even more than the actual build time. We parallelized it and time got reduced to 15 minutes. Now we are working at optimizing test startup time and data creation to even reduce the time. The saving is huge as it cuts the time for each Developer and enhances the overall Developer productivity.

Next thing to consider is how frequent your test run. If you run your Performance tests once in a week, then even if you notice a dip in performance it is really hard to pin point the actual commit message that caused the Performance to go down. If you can move it to daily, your chances suddenly are better to trace back such issues. Luckily many automation server tools help you orchestrate your jobs and you should view tools like Jenkins not only as Continuous Integration tools but as Continuous Testing tools.

The final comment on test execution is how the results are being reported. Most test execution results are raw and are in the shape of data. One thing that we did recently is to write series of Python scripts to manipulate that data and transform that into information. Such that results of our test execution are reported immediately on a Testing Dashboard in a Management friendly way like testing heat analysis . The end result is that at any time a Project Manager (or Product Manager or whoever is responsible for release decisions) can go and see what is the risk if we release just now.

How much emphasis you give to test execution phase? Have you tried something that is worth sharing with others?

Using Logs for Performance Testing

Performance testing is one of the specialty area for testers and is one of the challenging tasks. Designing Performance tests is different than designing functional tests, then setting up testing environment for Performance also needs special care and finally figuring out if the results do indicate an issue is not straight forward. One thing that has helped us lately in troubleshooting Performance issue is logging.

It is a known fact that logging is a Tester’s friend. Rather I’d say that it is a friend for any troubleshooter. The logs are like the traces which a skilled investigator follows to find the culprit who committed the crime. These are “Khurras” (Punjabi meaning foot prints) which “Khojis” (Punjabi/Urdu/Hindi meaning Detectors) follow. If the crime was committed (read an issue was observed) but no “khurras” are there (read no logs are there), it is not possible for the “Khoji” to find out the criminal (read what is the root cause of the issue).

(Original photo is here)

The first question in setting up logs is what logs we need? These have to be key points in software behavior for which time/memory captures are added to the log. For example in one of our Display systems where we implemented logging, we added logging for the following:

  • The time to open the file.
  • The time to display the thumbnails in the file.
  • The time to fill view with selected thumbnail.
  • The time to show properties when an element is selected in the display.

Once you know what points you need to log, the next step is to hook in logging. I’m hoping that your software or the tools/technologies that is being used by your software already has logging infrastructure in it and you have to add a new logger for Performance. And if it is unfortunately not there, then you need to have such infrastructure first. This logging will be added by the Programmers in your team and thus this project is another example of how Programmers and Testers can work together to deliver quality.

Suppose you have those friendly Programmers in your team and Performance logging is enabled. Now all you have to do is turn it on and run your existing tests. That’s it. Along with the numbers that measured before, you’ll get the logs.

Remember when all is going good, no one needs logs.

So you’ll need the logs only when your results are indicated a decline in Performance numbers that you measure. And then you can pass it on logs directly to the Programmers to investigate. Or if you are a smart tester like me, you’d write a small Python script to parse the log and get the meaningful numbers that you report to your team.

We’ve tried the above and believe it is really helping in issue investigation. We have moved through this from “We have poor Performance numbers” to “The Performance is down because file goes through up gradation progress while opening” like concrete issues being reported and fixed. Try it and you’ll love it. It will take some effort from both the Programmers and Testers to do it but it is surely worth an effort if you are serious about Performance testing.

Do you use logging in Performance testing? Or you have some other interesting ways to investigate Performance issues?

3 Misconceptions about Testing Career

Last week I was invited to speak to graduating batch of BS Computer Science at FAST Islamabad Campus. This was part of an initiative through which Industry professionals visit students and give them a true picture of real world. On my previous visits, I have talked about different aspects of Software Testing whereas this time I picked up misconceptions.

There is a long list of misconceptions but here were the 3 that I talked about:

  1. Don’t like Coding; Become a Tester.

That’s number one as many in Academia still believe Testing to be a non-coding job. That may have been true in 1990s when the profession of Software Testing was new and the gap was filled by people from other professions who were expert users and became testers. But the latest job market requires more and more testers who can write code to test code.

I shared examples of my employer’s recent hiring for Automation analyst and also shared SDET positions being announced in local market.

  1. Testers are paid less than Programmers.

This may have come in due to earlier practice of Industry to pay less to Testers who were Clickers in some respect. The pay scale of job reflects its complexity and it was fair to pay dumb testers less than Programmers. But again as the recent demand of smart and knowledgeable testers who are in some ways Gray Box Testers, the pay scale need to match that of a Programmer as both roles require intellectual work.

I used the survey data from P@SHA to share Entry level programmer salary vs. that of a Tester which suggests both are in the same league.

  1. Testers don’t grow. All important positions go to Programmers.

With testing becoming central to modern day Apps, testers are growing in multiple directions. There are testers who grow to run the entire testing department of the organizations. There are testers who grow to lead Engineering departments. And there are testers who become Testing experts in Performance testing or Automation or some other domain.

I used the Speakers list from our PSQC’17 that was held last month to give examples from each category. That really helped because local examples work more than any thing and I realized that holding a Software Quality Conference is helpful in many ways.

In the end there were many questions around the above myths and the realities involved. One question was aimed towards bitter relationship between Testers and Programmers for which I answered that this behavior needs to go and we need friendship between them to be able to deliver Quality Software.

What else you would like to add to this list of misconceptions?

Pakistan Software Quality Conference 2017

The first ever conference about Software Quality and perhaps the biggest Software Engineering conference in Pakistan’s history happened last weekend. Called as PSQC17 and organized through the platform of PSTB (Pakistan Software Testing Board), it drew ~275 Software Quality practitioners and researchers across the country.

This was a dream come true for all of us who have been building the community over the last three years through various events. And what a rewarding manifestation of our dream it was on the day. The registration desk opened at 8:30 AM and some testers were there already. Within next half an hour of Registration & Networking time, the beautiful Crystal Ball Room at the Marriott, Islamabad were filling rapidly. The proceedings started with recitation of Holy Quran and then the host of the event Roshan Masood from Confiz Ltd, Lahore took over the stage. With a smiling welcome to all and explaining the Conference theme “Testing in the Real”, Roshan invited President PSTB Dr. Zohaib Iqbal to give the opening note.

Dr. Zohaib first explained why talking about Software Quality is so important in today’s complex world which runs on software. He introduced the PSTB and it’s different objectives that include building a community and sharing knowledge. He briefly recapped the journey so far and how this Conference is a perfect climax to all those activities. Dr. Zohaib also shared demographics of the audience which included participation from all the provinces, almost one third audience representing female population and a mixed audience with varying experience. He set up some expectations for the enthusiastic gathering in the hall and laid down structure of the talks coming their way.

(more photos coming soon to facebook event page)

On this occasion, we were honored by the presence of Ambassadors in Pakistan from Portugal, Tunisia, Ukraine, Bulgaria and guests from Japan and China Embassy. Taking advantage to get in touch with top IT guys of the country, Portuguese Ambassador Joao Paulo Sabido Costa Addressed and shared how IT industry from both countries can work together for collective success. The Tunisian Ambassador Adel Elarbi also welcomed all the participants and extended his support in building the bridges between skillful people from both countries.

Here the first session of making connections finished with some refreshments offered to everyone. The testers from Lahore and Karachi were warmly welcomed by the ones from twin cities of Islamabad and Rawalpindi. I saw many new connections being made there which I can assure will go a long way.

The proceedings resumed through a talk on Quality Today by Kamran Shaukat Ali Khan, AVP Systems Ltd. who traveled from Lahore for the event. Kamran’s talk revolved around the basic questions of what Quality is, how we define it, what it costs to produce Quality and why we still see so many failures. Through references to classic materials like Quality is Free by Crosby and Deming’s principles, he involved the audience by asking many questions. He summed up by motivating the Testers to step up and take overall ownership of Quality in the products and solutions they deliver.  You can get the whole presentation here: PSQC17 – Quality Today – Kamran Shaukat

After this talk, we had to split the session in parallel. Many were not happy as they wanted to attend all the sessions and they were reminded that “Life is parallel and so is learning”.

At this point sessions were held back to back both in the main Conference Hall and the Ambassador Room. I moved around all the places to catch maximum on talks but some of my notes are based upon observations from others who attended the sessions.

In the main hall, next talk was on 5 Cs of DevOps by Syed Azhar ul Islam, Director DevOps Careem who flew from Karachi specially for the event. Azhar explained that to support products like Careem, the silos of Development, Testing, Operations, Product Management etc. need to be broken and all should work together. The summary slide had Planning drive Integration which drives Testing which gears Deployment which drives Monitoring & Feedback. All his slides are here: PSQC17 – 5Cs of DevOps – Syed Azhar ul Islam

We were being honored by so many guests in the session and I’ll just name a few Ather Imran (CEO Sybrid and President Open Islamabad), Kashif Moeen (CTO, Zigron), Dr. Amir Mahmood (Rector NU-FAST) and Dr. Mukhtar Ahmed from HEC.

In the other room, Syed Muhammad Afif Zaidi, Director Technology Pakistan S&P Global shared his journey of Building a QA setup from scratch as he has built a whole testing empire which he started as a lone tester. He gave many tips for the hiring process, building the teams, nurturing the talent, setting higher standards and constantly asking for more from the team.

Back in the main hall, next it was me talking about Testing without Specs. I started the talk by first sharing my personal experience so as how changing requirements which are outside our control make the Specification look like Specifiction (a term borrowed from Helena’s tweet). I then gave two tips to curb around it which were: talking to your team members to understand their mental maps and spending more time understanding the internals of the system you test. My last quote that “Devil is in the details, so is testing” got quite popular on the day. All the slides are here: PSQC17 – Testing Without Specs – Majd Uddin

This is the talk that I surely missed as it was parallel to mine though I so much wanted to sit there. Talking about Cost Effective Test Automation using Cloud  Adeel Shoukat (SQA Analyst) and Ehsen Raza (Senior SQA Analysst) from Contour Software Lahore, first shared the typical problem that each test automation project faces. Their experience along this journey came handy for the audience who could relate to this. They also showed small clips of the amazing work that their team is doing for others to get ideas and get inspired from. Their slides are here: PSQC17 – Cost-effective Test Automation Using Cloud Solution – Adeel Shoukat and Ehsen Raza

The last talk in the main hall before lunch was Open Source tools for DevOps testing by Adnan Maqsood Test Lead, LMKT, Islamabad. Adnan after spending some time explaining how testing fits into DevOps talked about the popular tools being used including Selenium, Appium, TestNG, Jenkins & Maven through some examples. His slides are here: PSQC17 – Open Source Tools for DevOps – Adnan Maqsood

In the other room, Shaima Niaz  Senior Software Quality Automation Engineer, Tkxel, Lahore was presenting Agile Test Automation. She started off with results of a survey on the subject conducted by her for the local Industry. One of the suggestion from her was to move to “Test and Dev” from “Test vs. Dev”. She then took examples from various recent concepts that help test and deliver products at fast pace including Industrial Automation, IoT testing and the concept of OTA (Over the Air). Her complete deck is here: PSQC17 – Agile Test Automation – Shaima Niaz

I’m sure that after reading so much new stuff, you need a break now. So you can imagine how the participants of PSQC17 were feeling after being filled with loads of information. Thus we decided to fill in the stomachs instead and use that time as an excuse to ask follow up questions from the Speakers. After that we resumed the session. Did I mention that Chicken Tikka Boti was very delicious.

Many media representative were attending the Confernece and they took these breaks as a good way to talk to the Speakrs and audience. PSQC17 were reported on at least 5 TV channels and you can watch one such report here.

It was some tutorial time after the break in the Ambassador Room where Amir Shahzad (Manager QA) and Hasan Farooq (Test Automation Engineer) from Stella Technologies, Islamabad presented Data driven testing using Spock Framework. Amir started the proceedings by explaining the concept of REST API testing and how data driven testing technique comes handy. He also shared his rational on why they picked Spock over other solutions. Hasan then showed some code examples of it’s implementation. There were many who liked this and were found discussing that they’ll try something similar in their own environments. The slides are here: PSQC17 – API Testing using Spock – Amir Shahzad and Hassan Farooq

In the meanwhile in the main hall, our own Dr. Uzair Khan from Quest Lab was sharing some real time implementation details of their research on Model Based Testing. He first maintained the stance that test automation is much bigger than just automating the test execution and if we can model the software behavior, we can achieve greater goals. He then shared some examples of using models to test popular video game Mario Bros and also example of using it in Embedded systems. His interesting slides are here: PSQC17 – Using Models to Automate Testing – Dr. Uzair Khan

Part of the PSQC17 agenda was to expand our collaboration with other domains and we had Nahil Mahmood come from Lahore to talk about Information Security as the last talk. Nahil who is CEO Delta Tech and is Chairman-designate of Pakistan Cyber Security Alliance, started off proceedings by sharing different reports depicting the current state of Information Security measures across the world and specially in Asia. He confirmed through multiple sources that situation in Pakistan is really bad and we need to take action on that. He then walked through al steps that can be taken for Software Security as part of SDLC what he called as sec-SDLC i.e. Secure SDLC. He summed up with “Software Quality includes Security. Let’s own it”. His slides are here: PSQC17 – Information Security in Pakistan – Nahil Mahmood

The Announcement from stage that #psqc17 is top trend of the day on the twitter made everyone happy and be proud of our social media team lead by Faiza Yousuf and supported by Saima Karim.

The afternoon nap was catching up everyone included me so we hurried towards a tea break. Also the participants gathered around the Registration desk one more time to collect their Certificates of Participation.

After this short break, we had a panel discussion on Future of Software Quality in Pakistan. The panelists were Shafiq Ahmed (Country Manager Stella Technologies Pakistan), Faiza Yousuf (Chief Consultatn at Outtabox who came from Karachi exclusively for this event) and Taimur Sarwar (Manager Operations at S&P Global Pakistan). They all shared their view of the future and then the hall was opened for questions. There were talks about how future students can be made more Quality conscious through modifications to the curriculu; there were discussion on why each of us need to step up and spend consistently on learning new technologies and techniques and there were many points to collaborate together to succeed together.

I was asked at the end to give a Closing Note in which I thanked through the core of my heart all Speakers, participants, our Sponsors. I then invited the whole organizing team (consisting of Dr. Zohaib Iqbal, Dr. Uzair Khan, Amir Shahzad, Hanan Atif, Farhana Adeel, Hina Zafar, Taimur Sarwar, Kashif Butt, Syed Qambar Ali, Zeeshan Asghar, Osman Ehsan, Haider Farooq) and all the volunteers who worked day in and day out to make this all happen. The team got a well deserved round of applause.

The final announcement was that PSQC17 is start of the journey, not the end and PSQC18 will be held in Lahore!

The feeling and emotions of everyone in the hall at the end were really great where they had a sense of belonging to this vibrant community, they were overwhelmed by such wonderful work being done in our own Industry and were committed to apply the ideas they learned in the day.

Long live Pakistan! Long live Testers from Pakistan!

Testers Meetup Lahore 3rd Edition

Lahore Qalandars may have been eliminated from PSL but the Lahore Testers have definitely proceeded to the next round of Software Testing excellence by gathering for the 3rd Meetup this weekend.

The venue this time was the beautiful FAST-NUCES auditorium and event was sponsored by Systems Ltd. through the platform of Pakistan Software Testing Board. That makes a really good combination where Industry and Academia are working together towards building the community. A special thanks to Kamran Khan who made it happen.

The event started with refreshing remarks by our host Ali Gohar. And quickly we were onto the first technical talk on the topic of “IoT Testing” by Khubaib ur Rehman from CTO247. Khubaib after explaining what we mean by Internet of Things and how complex is the world, talked briefly on different types of testing that are important in this domain. He expanded couple of testing types like Compatibility in terms of hardware variation and Performance Testing being dealing with the big data that comes in through those devices. Audience got engaged in the discussion and Ali Iqbal who is part of ICT R&D fund commented on the IoT Excellence Center being setup by ICT R&D. The slides are here: internet-of-things-testing


(more photos are here)

Khurram Bhatti who is based in Sweden and took advantage of this visit to get engaged with the Pakistani Testing community, then graced the stage and talked about “The real potential of Software Tester in Agile”. Through entertaining GIFs and some witty statements, Khurram set the tone on why testing takes the center stage in Agile world. He then shared his vision of there things that a Tester must work on to be successful in agile: Communication, Business and Technical. Khurram gave a nice example on how engine oil can be assumed as a Communication mechanism such that if the oil is not there, engine will not run log. The presentation is shared here: khurrambhatti_real-potential-of-tester-in-agile-world

Systems Ltd.’s representative Nauman Farooqui who is currently serving as VP, Project Delivery and Customer Satisfaction was invited to share his thoughts on “Producing Quality Software”. Nauman took this opportunity to share that he started his career as Software Tester and encouraged the Testers in audience to step up the game. He quickly recapped the illustrious ~40 years of journey of Systems Ltd. which is proudly the first Software House of Pakistan. The thing that I liked most on his suggestions to producing Software with top notch Quality was that the Testers should absolutely learn the Business domain they are serving. He also gave some recent examples on how talking to the Clients on regular basis results in more chances of success.

The final technical talk was on “Lessons Learned in Test Automation” by Adeel Shoukat from Contour Software. In his typical relaxed yet always to the point way, Adeel debunked some of the Automation myths and made it clear that “Record and Replay will never work”. He also strongly held his view that Automation will not replace the manual testing thus thinking of Automation as a way to reduce testing force is not a way to go. He then shared some examples from his own experience that building a framework based on a hybrid approach to solve your problem is a good way to adapt. He suggested to start simple and show the business value that Automation can do and show the value to the management that Automation adds. His slides are here: leasson_learned_in_automation

Dr. Zohaib Iqbal, President PSTB was invited to share all needed information for the upcoming Pakistan Software Quality Conference being held at Marriott hotel, Islamabad on April 8th. Dr. Zohaib recapped the whole Quality journey of these meetups that happened in Islamabad, Lahore and Karachi. And this demands for a climax like the upcoming Conference. Dr. Zohaib offered a special discounted offer to testers from Lahore and explained the “Call for Speakers” and “Sponsorship model”. More details about the Conference can be found on the website.

The best part of the event then happened. Yes the yummy food with tea but No, the delicious talks over cup of tea were more interesting. Testers were seen talking to the Speakers and hosts of the event to ask the questions which couldn’t be asked during the session; Testers were seen seeking guidance on their technical and people management issues; Testers were seen discussing ideas on how we can make these events more successful. These are just a few highlights of the over one hour talks that happened and I am still feeling awesome about them.

That’s all from the Lahore meetup and we are looking forward to see all of these brilliant testers join us at the Conference!