Do you do Testing Heat Analysis?

I’m sure that you do it in some form or other with may be a different name. And if you don’t do it, I think you should.

The idea is not wholely mine but this particular adaption is mine which I have been using for many years. The model looks good to be shared with other testers to use. it has some inspiration from Google’s ACC (Attributes Components Capability) model, and I have explained earlier too but to answer a specific question only.

Okay, this is what you should do:

  1. Divide your testing project in parts. The best count is 9 to help you draw the result in 3 x 3 shape. On what basis you can divide: if there are modules, or components, or facets, or attributes, or keywords or whatever works to help make decisions. The below example I have shows 9 core components in the Platform that we test.
  2. Decide on a color coding to show the “state of testing”. Testers also use happy, sad or angry faces to reflect on the fact how good is testing for a particular component, but colors help look it like a heat analysis. As if you expose this 3 x 3 cube to heat, how different parts will look after some exposure. The common color scheme is what you already know: Red for alarming situations, Yellow for cautions and Green for all good.
  3. You are good to go and do the analysis. You’ll need some definition on why you are marking a box as Red vs. marking it Green. Make sure you have both data and feelings to support it.

This is how it might look which uses an Orange color too:


So how this helps. In many ways:

  • As appeared in my original article about three years ago, it helps you where to place the new tester. Obviously you are concerned on the Red areas which you want to make green.
  • To make release decisions. How important are Red boxes for the current release? Remember, you’ll never get all Greens, so you have to make that decision on what is your acceptance criteria.
  • As a measure of progress for example, we share this heat analysis to reflect on functional code coverage for the components. I know those numbers are not the only thing to look for, but you still need some metrics to guide you if you are on track.

Have you tried this or similar models before? What did you find out?

Automation is not automating manual tests

The first thought that comes to a mind that doesn’t understand Testing and Automation is this: “I’ll automate all tests that are being executed manually and it will save time”.

This is what I thought almost a decade ago and added a column in the test plans that I used to execute. That column was named something link “Automated Test Id” and I started assigning Ids to each test that I automated and cross referenced it here. Looks nice, isn’t it.

Soon cracks appeared in my proposed system. There were some tests that I couldn’t automate e.g. the ones that needed visual verification of Engineering components. There were some automated tests that when I wrote, I added additional checks than what the manual test proposed e.g. checking more negative testing scenarios as it was easy to do that in automated tests compared to manual tests. Then there were new tests that I started to think of while running either automated or manual tests. I wasn’t sure if I should first write a test script for them and then add it’s entry in the test plan sheets OR I should first add it to my manual test plan and then automate it. This was one of the problems where I started to seek help from the internet and I concluded that my strategy was flawed. But I learned the following:

Manual tests has no correlation with automated tests. If they appear same sometimes, that is just a coincidence.

I also thought that anyone who has spent any serious effort in test automation would know that. But last week I saw the following in a project being managed in Microsoft’s Team Foundation Server:


It was okay to have this dialog in 2006 but towards the end of 2016, it is a sin. I’m not sure if this is standard one coming from Microsoft or it was done by some geek who manages the project.

Joel Spolsky said around that time in this context of automating all tests:

And so one result of the new emphasis on automated testing was that the Vista release of Windows was extremely inconsistent and unpolished. Lots of obvious problems got through in the final product… none of which was a “bug” by the definition of the automated scripts, but every one of which contributed to the general feeling that Vista was a downgrade from XP. The geeky definition of quality won out over the suit’s definition; I’m sure the automated scripts for Windows Vista are running at 100% success right now at Microsoft, but it doesn’t help when just about every tech reviewer is advising people to stick with XP for as long as humanly possible.

When people have the above-mentioned thought process, they say/ask the following things. I’d suggest to stay away from these:

  • Our testing is 75% automated. See this is because they believe that we have this many tests in total and out of which, that many are automated.
  • Through automation we have saved 250 hours. Which means that if those tests were executed manually for this many times, such would be the time taken to accomplish this.
  • By automating all our testing, we’ll downsize our testing team. Because they will replace the manual tests being executed by those testers.
  • Which manual tests I should be automating?

The above statements keep coming because a) the tool vendors create this hype as they want to sell their phakkis and b) the management who runs the show has minimal understanding of what testing is and what automation is.

Have you said or seen above understanding on test automation? Do you believe in them or have any thoughts on this?

“Defense in depth” testing strategy

There is nothing that we can call “excessive testing”. The more you test, the better it is.

When we get to know the first few Laws of Testing and understand that “Testing can never be complete”, we get into the mode of limiting our testing to the constraints. This often includes selecting the best tests to execute in given time, leaving out some testing types and worse just having breadth in testing for the sake of leaving all the depth in testing.

Last week, I was discussing with my manager on the possible approaches we can take to add tests for multi-threading in our unit testing suite. He is based in the US, leads a big development team and has done many successful releases of complex systems, all approaches we discussed had some flaws and we couldn’t agree on the “best approach”. You can guess the decision we made. No, it was not to drop this type of testing altogether because we don’t know the best way to do that. The decision was to try the approach that is one of the candidates of becoming a good approach, do it for a while and learn from it.

That is what my manager described as a test strategy to have “Defense in depth” where you can imagine a castle which is built on the top of a hill with a big wall and then outside fence and then there are trenches and then there are landmines around it and then there are other ways to protect the castle. All of these layers of defense have some advantage over the other and though they cannot protect from all type of attacks, they still limit the chance of attack by capturing some of it.

Or you can imagine a soccer team that has the Italian strategy to never allow the opposition to score a goal.


(the image is taken from here: )

I have often shared my views about Unit tests being a safety net which allows Programmers to make changes without fear that if they fall, they’ll not fall on the ground. So in that respect, why just have one safety net, why not have multiple nets such that if the fall is big and it raptures the first net, the other one saves you from falling on the ground.

In my experience, the following layers of protection or safety nets are kind of a must:

  • Set of Unit tests that run with the individual code manipulations. They are by default automated and usually executed multiple times in a day.
  • Set of tests that are executed immediately after the build is made. These are Smoke tests and other tests such as Performance which were not executed as part of the build. These should also be automated tests and run with each build.
  • Set of tests that are Exploratory in nature and are executed by Testers who have knowledge of the domain and the overall system. Some experts believe that these tests are useless or repetition if we have the above two layers but as I mentioned above, they add another layer of defense. And I have seen many big bugs coming from this round of testing as the above two are merely Regression tests.

Do you believe in having depth in your software testing layers? How many layers you have or would like to have?

Testers Meetup Activity reaches Karachi

Karachi known as commercial capital of Pakistan has an iconic image such that if it doesn’t happen here, it doesn’t exist. As the activity based Testers Meetup met the vibrant and passionate Testers community in Karachi last Saturday after having successful rounds in Islamabad and Lahore, we can safely say that such activities are here to stay.

The event was well orchestrated by our partners VentureDive  team gyrating around testing champion Zeeshan Asghar. The event was running under the banner of Pakistan Software Testing Board (PSTB) . The cafeteria of Dot Zero  located at 14th floor at the famous Tariq Road was a perfect choice for the event where ~50 testers were invited to attend the event. The host Mohsin Ali started the proceedings once the teams were settled on their tables.

The event was kicked off by an opening note by Usman Zuberi who is Project Manager at VentureDive. Usman welcomed all on behalf of his organization and recapped a brief yet rising journey of the company. His message to the testers was very clear: “Don’t adapt this career just because you think you are misfit as a Programmer.” Usman affirmed the audience that if they take testing career as a passion, they have a long road of success ahead of them.

The day was about the participants broken into 9 teams doing live rounds of testing. The fiercely fought competition went through the stages of planning the tests, executing the tests, finding the bugs and finally presenting the findings to the audience. To keep with the tradition of so many TV shows running from Karachi, there were cash prizes for the top three teams. The application that was given to the teams was a typical eCommerce site with both customer and admin modules. The teams took part in each round with lot of energy and it was heartening to see so many brilliant testes showing their skills in action.


(More pictures are on our facebook page)

Since knowledge comes from doing things but it also comes from learning from people who have already done those things, the above rounds of testing were sliced through some knowledge sharing sessions.

Syed Azhar , Director DvOps at Careem explained how modern day apps testing can’t limit itself to just functional testing. By giving examples from the scope and depth of the architecture that Careem utilizes to make it a success, Azhar said that living in silos of Business Requirements, Product Management, Development, Testing, Deployment are not the way to go. DevOps brings all of them together and he also explained the 5 Cs of DevOps. His full presentation is here:  devops-solution-qa

The most loved talk of the day was on lessons learned in Test Automation by Syed Tassaduq Hussain  who now heads the Quality Assurance team of TPS. In his typical style, Tassaduq shared many rules through his personal experience where he split the lessons amongst Management and Engineers. One of his suggestion to the test management was to form the team such that it benefits maximum automation ROI because automation cannot be done by the manual testers on their spare time. For Engineers, he explained what Flaky tests are and how we can conquer them to build confidence in automation. The slides are here: automation-testing

The third talk was by our own Zeeshan Asghar on the challenges we face in the testing where so many platforms, so many devices and so many browsers are out there where Compatibility and Regression testing has to be run. Zeeshan suggested to use available tools to accomplish this so that testers focus on tasks that require human intelligence and give such routine work to the testing platforms out there. See more details here: software-trends

The event was also graced by two Meetup pioneers from Islamabad, Amir Shahzad who is Manager Testing at Stella and Nouman Umer representing Zigron. Amir shared his feelings on how such meetups help testers grow together and was happy to see that these forums are now reaching new locations.

Taking this opportunity, I briefly talked about what PSTB is, the certifications it offers and how it’s objectives are well aligned with the needs of Pakistan Testing community. I shared the formula of success:

Testers Meet + Learn together in fun setup + Build Connections = Testers Succeed

Also invited the testers to the proposed National level Testing conference which we plan to do in March 2017 Insha Allah (God Willing). The slides are here: pstb-and-meetups

The event was also graced by Asim Kazmi, leader of 10Pearls testing and who is one of the faces of community. Another popular figure Faiza Yousuf couldn’t join us but gave us full support and some selected University students of her joined us in what we calling making the future testers.

It would be an injustice to not name all the volunteering team that included Shahan Jahangir, Laraib Vaqar Patoli, Mohsin Ali, Mariam Hammad and Shafaq Zehra (pardon me if I missed any one) who made all the proceedings smooth. And did I forget to say that the “Biryani” lunch was lovely 🙂

Thanks to all the participants in joining us in the journey of Testers Excellence. Let’s work together to take it to the next level.

Agile Conference Pakistan 2016 – Takeaways

One of the sources of knowledge is to know the sources of knowledge. Each society has sub-societies in it and if you are not there, you’d think that this never happens in our city. This exactly is the case for me but I was lucky to become part of Pakistan Agile Development Society last year through the 2nd edition of Agile Conference Pakistan. That allowed me to learn more throughout the year and the best part came in the last weekend when along with a wonderful team of organizers, speakers, sponsors, volunteers and audience I enjoyed the Agile Conference Pakistan 2016  which was attended by ~250 Agile practitioners.

Below is a very very short summary on what happened there and more will be shared on Society’s facebook page and website. The first set of photos is out and soon the slides and recordings will also be shared.

Society’s chairman Suhail Iqbal said in opening note that it is a very happy trend where more and more people are joining the Agile conference. He also mentioned that though Agile movement arrived a little late in the Pakistan IT industry but we can work together to make it happen in a great way.


The first talk was “Agile Transformation vs. Agile Adaption” by Syed Ahmed who is CEO of DPL-IT and ex-chairman of P@SHA.  Syed started the talk with difference between Process-centric companies and People-centric companies and rejected the notion that one process can change the world rather it is the human potential that changes the world. With some examples he explained that the yardstick of whether a company is process or people-centric is the answer to this question: “Do you allow people to make mistakes?” His view was that Agile Adoption is like doing the rituals whereas a Transformation will come when you really have belief in something. After establishing these facts, he presented the case study of his own company DPL-IT where they have gone through an Agile Transformation. He shared details on how they are promoting a “Rebel ethos” culture and how they have a “Sir-carr” which fines you for a certain money if you call people as “Sir” rather than with their names. He also suggested a flipped hierarchy as below (this is drawn by me and not the actual slide):


In the last few slides, he had some lessons learned which included a “Let Go” thinking in top management’s mind, making things transparent and having a belief that the results will start coming though there will be some rough patches.

The next talk was about some suggested recommendations on Managing Agile Transformation by Javad Ahmed who is COO of a US based company Oracular (Smart-IS in Pakistan). Javad gave his viewpoint on what Agile is, why we need it, and how he has implemented it in his own scope. Given his consultancy background, through various examples he shared the practical implications of doing an organization wide Agile transformation. He had lot of good advice to offer for bringing in the “business perspective” in making all decisions and below are some quotes that I noted:

  • Nothing can be done without Top management’s commitment and they’ll only buy-in if they see business value.
  • Agile is focused on “Right Results” and an “On-target” outcome.
  • Agile is an ideology.
  • Agile facilitates prioritization.
  • Do not short-change staffing e.g. asking a Scrum Master to play as Product Owner too.
  • Counselling and Coaching is very important in the Agile journey.
  • Too many times the documentation is like a solved Pakistan Studies papers that happen in our education system where marks are given based on how lengthy and colorful is the answer.

Then Afif Zaid and Mujeeb Zahoor shared the story of Agile success at S&P Global’s Pakistan office. Mujeeb walked through a brief history of the overall organization and the stuff that happens in Islamabad. Afif then provided inside scoop of how they worked through this Agile Transformation phase and how they reached to the milestone of having 57 successful SCRUM teams across the globe. His tips for others to do the same were passing on the vision, help team on setting the priorities and making teams self-organized and empowered. One of the many experiments they are doing is playing a “Shark Tank” to allow everyone to throw ideas. Mujeeb then concluded the talk by sharing the business success that Agility has brought to them.

OPEN Islamabad’s President and CEO of Sybrid, Ather Imran was the first speaker after lunch to talk about “Agile Culture and Knowledge Organizations in Pakistan. Ather referred to Peter Drucker’s work on what a Knowledge Worker is and what is a Knowledge Organization. Then after briefly going through the Agile principles, he concluded that:

Knowledge Organization  = Agile

He then brought in some interesting thoughts on the Power Psychology and how we tend to be followers by default. Through examples including the South Korean Airline experience from Malcolm Gladwell’s book Outlier, Ather made it clear that our culture is where people defer to the Power and want to be obedient to the authority. He then shared a few tips to break that with consistent efforts which included 10 Rules that he explained one by one. For example he wants to “Delegate to the level of Anarchy” because if you are delegating and are comfortable with that, there is still control in your hands. So getting uncomfortable with delegation is the key.

The next talk was by the leadership training maestro Mohsin Lodhi who talked about “Leadership in the Digital World”. Mohsin started the talk with establishing the fact that how technology is eating many jobs and many industries are now squeezed to hand held devices. Through his typical style of backing statements with numbers, he shared many new researches that enforces the reality that we are living in a fast paced and non-linear changing world. He connected the dots that Results are very important but one should never forget that the results come through processes and processes are due to a culture and creating a culture is the job of the leader. (The below is my notes and not actual slide)


He then through some moving and dancing slides, shared the lessons from the famous social experiment done with 5 monkeys  and suggested not to react to the shower of culture around us by saying that “that’s how it has always been”. At the end he also explained the Change Equation and how leaders should work on the buy-in and show through their actions if they are serious about bringing any change.

An Agile Coach at VizTek and an Agile lover Nabeel Ansar then talked about “Avoiding common mistakes in Agile Transformations”. Nabeel started the talk with when and how Agile fails and emphasized on the conditions in which you apply Agile to be more responsible for the failure. He gave an interesting example of driving a Ferrari on the IJP road and then blaming the road for the poor performance of the car. Through his personal experience of Agile Transformation in few organizations, he recommended to first “show something by doing it” and then ask for others to help. He also shared 8 Agile Transformation Patterns which included the likes of “Start small” and “Seed and Grow”. The point that I most liked in the talk was Nabeel’s insistence on improving the Engineering practices because a fast delivery is only possible if you have the automation, testing and other build processes geared towards such delivery.

A short closing note was delivered by Farid Zafar, Associate Professor at Bahria University Islamabad thanking the audience for sparing their precious weekend day for the love of Agile.

Believe me my notepad had much more than the above and I’m sure that was the case with all the participants. Let’s keep this journey towards excellence together and I hope to share some after thoughts soon on some of the subjects listed here!

Four years of blogging

Knowledge Tester turned 4 earlier this month.

This has been an amazing journey during which I have made so many new online and real life friends along with learning so much new stuff which could not have been possible without having a platform like this. I was lucky to have started it at the right time in my career and will continue my efforts to make it even bigger.


(The image is taken from: here)

As I recapped first, second and third years, I picked the highlights of the year being celebrated, let me do the same this year. In those terms if I look at the fourth year, it was a year of expansion where follower-ship of blog increased at a pace of 3x compared to first year. Also per blog viewership doubled this year which indicated that the blog has more penetration now compared to the past.

But all these are kind of personal milestones and the thing that I most enjoyed this year was the fact that the Pakistani testing community is finally waking up as a results of this effort. We had back to back testers meetup in Islamabad and Lahore this year and have a plan to do one in Karachi later this year. And the dream of first Testing Conference in Pakistan seems so real that we have a tentative date in the month of March 2017 locked for the event. I’ll be sharing more details in coming weeks so that nobody misses it out.

What this meant was that about 60% of my blog posts in fourth year was either a report of an event or a follow up discussion on what happened at the event. This reflects the picture really well that Knowledge Tester is no longer just a platform to promote a single person’s vision rather it is a platform for the testing community to share.

As all of this happened, others are joining the party which makes me so happy. You know that you are influential when people start doing what you do without knowing why they are doing. You know that you are influential when companies hire and pay people to do the job that you are doing voluntarily. And you know that you are influential when people start taking the path defined by you without knowing who is leading this.

Before I talk about the plans in the fifth year, the last thing to summarize last year is to invite you to have a look at the most popular blog post of the year: Smoke testing is boring, Help!!!

Well, the top priority for the fifth year is have the testing conference happen where we could host testers from all over Pakistan. I need lot of help and support from all the gang and I know you’ll not disappoint me. Other than that, I am thinking to either record some video sessions or write a series of posts which serve as a course for people who are new to the profession of software testing. There are many helpful material out there but we still need one that covers this subject from a local perspective. Let’s see how much of that I can achieve.

Thanks for all your wonderful support in these years. What topics you would like me to cover this year?

Generalist, Specialist, T and Pi shaped Testers

“All testers are special but some are more special than others”. To borrow a line from Animal Farm with some variation.

In last month’s testers meetup, one of the discussion points was “Should testers be Generalist or Specialist?” As the discussion was happening, I decided to write a blog post on this subject as it is close to my heart. It took me almost a month to do that which is bad given the fast paced world we live in but not as bad if compared with the whole span of life.

I thought I’ll start with an overview of what we mean by Specialist and Generalist etc. but my draft was in progress when I saw this excellent post at Testastic blog which I think lays a good foundation for the readers. Also as I was researching this topic, I found this great webinar by Derk-Jan de Grood and Jan Jaap Cannegieter done for EuroSTAR which introduces T and Pi shaped very well. My advice would be to leave my blog here and first go to the above links.

Thanks to the sincere readers of the blogs for consulting the references and welcome back. To all other casual readerrs, T shaped professional is a person who has a depth of knowledge on a Specialty but has enough breadth of knowledge on the General subject. A typical example that I give is from medical profession i.e. an ENT specialist while recommending a medicine needs to be aware if the patient has heart disease or is pregnant to avoid any side effects. A Pi shaped professional is then the one who has two special legs meaning they are specialists on two subjects. See the picture below that I copied from the webinar slides that I mentioned above.


(Slide from webinar: )

With that background, let’s move to the topic now.

First and foremost, a Tester is already a T shaped professional. If not, tester should be. Testers are members of the team who should have enough knowledge of different functions of the teams like Development, Deployment, Build process, Documentation, Release management etc. yet their specialty lies in Testing. The teams are aware of that and they usually respect that by leaving the testing job to the Testing Specialists.

Some people gets confused with the cross-functional aspect of Agile teams and say that all members test so how come a Tester is special. But good teams know that having a Specialist tester help. In one of the teams that I joined last year, the project lead said to me:

Majd, I don’t want you to spend most of the time writing tests. I’d like to spend most of your time to see how we are doing on the testing front.

So if you are a Tester, you are already a Specialist on testing and you should rather be a T shaped professional by learning the other functions of the team.

The question then comes to developing the other specialty or second leg of a Pi shaped testers. In the webinar mentioned above, when the presenters did few rounds of workshops with testers they found out that the second leg can be Programming, Management, Automation etc. All of those are good but in my opinion for a Tester, knowledge of other fields like Programming or Management actually makes them a better T shaped Tester by expanding breadth of knowledge. A Pi shaped tester’s other leg should also be in testing.

I don’t want to force my opinion and you are free to pick second legs other than a Testing area. But if you take my advice, I’d suggest the other legs could be Performance testing, Security Testing, Usability Testing, Automated Testing etc. By that I mean that you should be subject matter specialist on the subject and not just a Generalist on all. If you want to get started, here is a piece from tester friend Rizwan Jafri that came out this week about automation.

Are you a T shaped or Pi shaped tester? If not, at what stage you are. And if yes, what are the second leg options you’d like to share with us.