“Defense in depth” testing strategy

There is nothing that we can call “excessive testing”. The more you test, the better it is.

When we get to know the first few Laws of Testing and understand that “Testing can never be complete”, we get into the mode of limiting our testing to the constraints. This often includes selecting the best tests to execute in given time, leaving out some testing types and worse just having breadth in testing for the sake of leaving all the depth in testing.

Last week, I was discussing with my manager on the possible approaches we can take to add tests for multi-threading in our unit testing suite. He is based in the US, leads a big development team and has done many successful releases of complex systems, all approaches we discussed had some flaws and we couldn’t agree on the “best approach”. You can guess the decision we made. No, it was not to drop this type of testing altogether because we don’t know the best way to do that. The decision was to try the approach that is one of the candidates of becoming a good approach, do it for a while and learn from it.

That is what my manager described as a test strategy to have “Defense in depth” where you can imagine a castle which is built on the top of a hill with a big wall and then outside fence and then there are trenches and then there are landmines around it and then there are other ways to protect the castle. All of these layers of defense have some advantage over the other and though they cannot protect from all type of attacks, they still limit the chance of attack by capturing some of it.

Or you can imagine a soccer team that has the Italian strategy to never allow the opposition to score a goal.

two-wall-defense1-570x301

(the image is taken from here: http://blog.smartbear.com/design/what-medieval-castles-can-teach-you-about-web-security/ )

I have often shared my views about Unit tests being a safety net which allows Programmers to make changes without fear that if they fall, they’ll not fall on the ground. So in that respect, why just have one safety net, why not have multiple nets such that if the fall is big and it raptures the first net, the other one saves you from falling on the ground.

In my experience, the following layers of protection or safety nets are kind of a must:

  • Set of Unit tests that run with the individual code manipulations. They are by default automated and usually executed multiple times in a day.
  • Set of tests that are executed immediately after the build is made. These are Smoke tests and other tests such as Performance which were not executed as part of the build. These should also be automated tests and run with each build.
  • Set of tests that are Exploratory in nature and are executed by Testers who have knowledge of the domain and the overall system. Some experts believe that these tests are useless or repetition if we have the above two layers but as I mentioned above, they add another layer of defense. And I have seen many big bugs coming from this round of testing as the above two are merely Regression tests.

Do you believe in having depth in your software testing layers? How many layers you have or would like to have?

Testers Meetup Activity reaches Karachi

Karachi known as commercial capital of Pakistan has an iconic image such that if it doesn’t happen here, it doesn’t exist. As the activity based Testers Meetup met the vibrant and passionate Testers community in Karachi last Saturday after having successful rounds in Islamabad and Lahore, we can safely say that such activities are here to stay.

The event was well orchestrated by our partners VentureDive  team gyrating around testing champion Zeeshan Asghar. The event was running under the banner of Pakistan Software Testing Board (PSTB) . The cafeteria of Dot Zero  located at 14th floor at the famous Tariq Road was a perfect choice for the event where ~50 testers were invited to attend the event. The host Mohsin Ali started the proceedings once the teams were settled on their tables.

The event was kicked off by an opening note by Usman Zuberi who is Project Manager at VentureDive. Usman welcomed all on behalf of his organization and recapped a brief yet rising journey of the company. His message to the testers was very clear: “Don’t adapt this career just because you think you are misfit as a Programmer.” Usman affirmed the audience that if they take testing career as a passion, they have a long road of success ahead of them.

The day was about the participants broken into 9 teams doing live rounds of testing. The fiercely fought competition went through the stages of planning the tests, executing the tests, finding the bugs and finally presenting the findings to the audience. To keep with the tradition of so many TV shows running from Karachi, there were cash prizes for the top three teams. The application that was given to the teams was a typical eCommerce site with both customer and admin modules. The teams took part in each round with lot of energy and it was heartening to see so many brilliant testes showing their skills in action.

karachimeetup

(More pictures are on our facebook page)

Since knowledge comes from doing things but it also comes from learning from people who have already done those things, the above rounds of testing were sliced through some knowledge sharing sessions.

Syed Azhar , Director DvOps at Careem explained how modern day apps testing can’t limit itself to just functional testing. By giving examples from the scope and depth of the architecture that Careem utilizes to make it a success, Azhar said that living in silos of Business Requirements, Product Management, Development, Testing, Deployment are not the way to go. DevOps brings all of them together and he also explained the 5 Cs of DevOps. His full presentation is here:  devops-solution-qa

The most loved talk of the day was on lessons learned in Test Automation by Syed Tassaduq Hussain  who now heads the Quality Assurance team of TPS. In his typical style, Tassaduq shared many rules through his personal experience where he split the lessons amongst Management and Engineers. One of his suggestion to the test management was to form the team such that it benefits maximum automation ROI because automation cannot be done by the manual testers on their spare time. For Engineers, he explained what Flaky tests are and how we can conquer them to build confidence in automation. The slides are here: automation-testing

The third talk was by our own Zeeshan Asghar on the challenges we face in the testing where so many platforms, so many devices and so many browsers are out there where Compatibility and Regression testing has to be run. Zeeshan suggested to use available tools to accomplish this so that testers focus on tasks that require human intelligence and give such routine work to the testing platforms out there. See more details here: software-trends

The event was also graced by two Meetup pioneers from Islamabad, Amir Shahzad who is Manager Testing at Stella and Nouman Umer representing Zigron. Amir shared his feelings on how such meetups help testers grow together and was happy to see that these forums are now reaching new locations.

Taking this opportunity, I briefly talked about what PSTB is, the certifications it offers and how it’s objectives are well aligned with the needs of Pakistan Testing community. I shared the formula of success:

Testers Meet + Learn together in fun setup + Build Connections = Testers Succeed

Also invited the testers to the proposed National level Testing conference which we plan to do in March 2017 Insha Allah (God Willing). The slides are here: pstb-and-meetups

The event was also graced by Asim Kazmi, leader of 10Pearls testing and who is one of the faces of community. Another popular figure Faiza Yousuf couldn’t join us but gave us full support and some selected University students of her joined us in what we calling making the future testers.

It would be an injustice to not name all the volunteering team that included Shahan Jahangir, Laraib Vaqar Patoli, Mohsin Ali, Mariam Hammad and Shafaq Zehra (pardon me if I missed any one) who made all the proceedings smooth. And did I forget to say that the “Biryani” lunch was lovely🙂

Thanks to all the participants in joining us in the journey of Testers Excellence. Let’s work together to take it to the next level.

Agile Conference Pakistan 2016 – Takeaways

One of the sources of knowledge is to know the sources of knowledge. Each society has sub-societies in it and if you are not there, you’d think that this never happens in our city. This exactly is the case for me but I was lucky to become part of Pakistan Agile Development Society last year through the 2nd edition of Agile Conference Pakistan. That allowed me to learn more throughout the year and the best part came in the last weekend when along with a wonderful team of organizers, speakers, sponsors, volunteers and audience I enjoyed the Agile Conference Pakistan 2016  which was attended by ~250 Agile practitioners.

Below is a very very short summary on what happened there and more will be shared on Society’s facebook page and website. The first set of photos is out and soon the slides and recordings will also be shared.

Society’s chairman Suhail Iqbal said in opening note that it is a very happy trend where more and more people are joining the Agile conference. He also mentioned that though Agile movement arrived a little late in the Pakistan IT industry but we can work together to make it happen in a great way.

acp2016

The first talk was “Agile Transformation vs. Agile Adaption” by Syed Ahmed who is CEO of DPL-IT and ex-chairman of P@SHA.  Syed started the talk with difference between Process-centric companies and People-centric companies and rejected the notion that one process can change the world rather it is the human potential that changes the world. With some examples he explained that the yardstick of whether a company is process or people-centric is the answer to this question: “Do you allow people to make mistakes?” His view was that Agile Adoption is like doing the rituals whereas a Transformation will come when you really have belief in something. After establishing these facts, he presented the case study of his own company DPL-IT where they have gone through an Agile Transformation. He shared details on how they are promoting a “Rebel ethos” culture and how they have a “Sir-carr” which fines you for a certain money if you call people as “Sir” rather than with their names. He also suggested a flipped hierarchy as below (this is drawn by me and not the actual slide):

flippedhiearchy

In the last few slides, he had some lessons learned which included a “Let Go” thinking in top management’s mind, making things transparent and having a belief that the results will start coming though there will be some rough patches.

The next talk was about some suggested recommendations on Managing Agile Transformation by Javad Ahmed who is COO of a US based company Oracular (Smart-IS in Pakistan). Javad gave his viewpoint on what Agile is, why we need it, and how he has implemented it in his own scope. Given his consultancy background, through various examples he shared the practical implications of doing an organization wide Agile transformation. He had lot of good advice to offer for bringing in the “business perspective” in making all decisions and below are some quotes that I noted:

  • Nothing can be done without Top management’s commitment and they’ll only buy-in if they see business value.
  • Agile is focused on “Right Results” and an “On-target” outcome.
  • Agile is an ideology.
  • Agile facilitates prioritization.
  • Do not short-change staffing e.g. asking a Scrum Master to play as Product Owner too.
  • Counselling and Coaching is very important in the Agile journey.
  • Too many times the documentation is like a solved Pakistan Studies papers that happen in our education system where marks are given based on how lengthy and colorful is the answer.

Then Afif Zaid and Mujeeb Zahoor shared the story of Agile success at S&P Global’s Pakistan office. Mujeeb walked through a brief history of the overall organization and the stuff that happens in Islamabad. Afif then provided inside scoop of how they worked through this Agile Transformation phase and how they reached to the milestone of having 57 successful SCRUM teams across the globe. His tips for others to do the same were passing on the vision, help team on setting the priorities and making teams self-organized and empowered. One of the many experiments they are doing is playing a “Shark Tank” to allow everyone to throw ideas. Mujeeb then concluded the talk by sharing the business success that Agility has brought to them.

OPEN Islamabad’s President and CEO of Sybrid, Ather Imran was the first speaker after lunch to talk about “Agile Culture and Knowledge Organizations in Pakistan. Ather referred to Peter Drucker’s work on what a Knowledge Worker is and what is a Knowledge Organization. Then after briefly going through the Agile principles, he concluded that:

Knowledge Organization  = Agile

He then brought in some interesting thoughts on the Power Psychology and how we tend to be followers by default. Through examples including the South Korean Airline experience from Malcolm Gladwell’s book Outlier, Ather made it clear that our culture is where people defer to the Power and want to be obedient to the authority. He then shared a few tips to break that with consistent efforts which included 10 Rules that he explained one by one. For example he wants to “Delegate to the level of Anarchy” because if you are delegating and are comfortable with that, there is still control in your hands. So getting uncomfortable with delegation is the key.

The next talk was by the leadership training maestro Mohsin Lodhi who talked about “Leadership in the Digital World”. Mohsin started the talk with establishing the fact that how technology is eating many jobs and many industries are now squeezed to hand held devices. Through his typical style of backing statements with numbers, he shared many new researches that enforces the reality that we are living in a fast paced and non-linear changing world. He connected the dots that Results are very important but one should never forget that the results come through processes and processes are due to a culture and creating a culture is the job of the leader. (The below is my notes and not actual slide)

cultureandresults

He then through some moving and dancing slides, shared the lessons from the famous social experiment done with 5 monkeys  and suggested not to react to the shower of culture around us by saying that “that’s how it has always been”. At the end he also explained the Change Equation and how leaders should work on the buy-in and show through their actions if they are serious about bringing any change.

An Agile Coach at VizTek and an Agile lover Nabeel Ansar then talked about “Avoiding common mistakes in Agile Transformations”. Nabeel started the talk with when and how Agile fails and emphasized on the conditions in which you apply Agile to be more responsible for the failure. He gave an interesting example of driving a Ferrari on the IJP road and then blaming the road for the poor performance of the car. Through his personal experience of Agile Transformation in few organizations, he recommended to first “show something by doing it” and then ask for others to help. He also shared 8 Agile Transformation Patterns which included the likes of “Start small” and “Seed and Grow”. The point that I most liked in the talk was Nabeel’s insistence on improving the Engineering practices because a fast delivery is only possible if you have the automation, testing and other build processes geared towards such delivery.

A short closing note was delivered by Farid Zafar, Associate Professor at Bahria University Islamabad thanking the audience for sparing their precious weekend day for the love of Agile.

Believe me my notepad had much more than the above and I’m sure that was the case with all the participants. Let’s keep this journey towards excellence together and I hope to share some after thoughts soon on some of the subjects listed here!

Four years of blogging

Knowledge Tester turned 4 earlier this month.

This has been an amazing journey during which I have made so many new online and real life friends along with learning so much new stuff which could not have been possible without having a platform like this. I was lucky to have started it at the right time in my career and will continue my efforts to make it even bigger.

4-years

(The image is taken from: here)

As I recapped first, second and third years, I picked the highlights of the year being celebrated, let me do the same this year. In those terms if I look at the fourth year, it was a year of expansion where follower-ship of blog increased at a pace of 3x compared to first year. Also per blog viewership doubled this year which indicated that the blog has more penetration now compared to the past.

But all these are kind of personal milestones and the thing that I most enjoyed this year was the fact that the Pakistani testing community is finally waking up as a results of this effort. We had back to back testers meetup in Islamabad and Lahore this year and have a plan to do one in Karachi later this year. And the dream of first Testing Conference in Pakistan seems so real that we have a tentative date in the month of March 2017 locked for the event. I’ll be sharing more details in coming weeks so that nobody misses it out.

What this meant was that about 60% of my blog posts in fourth year was either a report of an event or a follow up discussion on what happened at the event. This reflects the picture really well that Knowledge Tester is no longer just a platform to promote a single person’s vision rather it is a platform for the testing community to share.

As all of this happened, others are joining the party which makes me so happy. You know that you are influential when people start doing what you do without knowing why they are doing. You know that you are influential when companies hire and pay people to do the job that you are doing voluntarily. And you know that you are influential when people start taking the path defined by you without knowing who is leading this.

Before I talk about the plans in the fifth year, the last thing to summarize last year is to invite you to have a look at the most popular blog post of the year: Smoke testing is boring, Help!!!

Well, the top priority for the fifth year is have the testing conference happen where we could host testers from all over Pakistan. I need lot of help and support from all the gang and I know you’ll not disappoint me. Other than that, I am thinking to either record some video sessions or write a series of posts which serve as a course for people who are new to the profession of software testing. There are many helpful material out there but we still need one that covers this subject from a local perspective. Let’s see how much of that I can achieve.

Thanks for all your wonderful support in these years. What topics you would like me to cover this year?

Generalist, Specialist, T and Pi shaped Testers

“All testers are special but some are more special than others”. To borrow a line from Animal Farm with some variation.

In last month’s testers meetup, one of the discussion points was “Should testers be Generalist or Specialist?” As the discussion was happening, I decided to write a blog post on this subject as it is close to my heart. It took me almost a month to do that which is bad given the fast paced world we live in but not as bad if compared with the whole span of life.

I thought I’ll start with an overview of what we mean by Specialist and Generalist etc. but my draft was in progress when I saw this excellent post at Testastic blog which I think lays a good foundation for the readers. Also as I was researching this topic, I found this great webinar by Derk-Jan de Grood and Jan Jaap Cannegieter done for EuroSTAR which introduces T and Pi shaped very well. My advice would be to leave my blog here and first go to the above links.

Thanks to the sincere readers of the blogs for consulting the references and welcome back. To all other casual readerrs, T shaped professional is a person who has a depth of knowledge on a Specialty but has enough breadth of knowledge on the General subject. A typical example that I give is from medical profession i.e. an ENT specialist while recommending a medicine needs to be aware if the patient has heart disease or is pregnant to avoid any side effects. A Pi shaped professional is then the one who has two special legs meaning they are specialists on two subjects. See the picture below that I copied from the webinar slides that I mentioned above.

tandpishaped

(Slide from webinar: https://www.youtube.com/watch?v=XzA-FvRG0DE )

With that background, let’s move to the topic now.

First and foremost, a Tester is already a T shaped professional. If not, tester should be. Testers are members of the team who should have enough knowledge of different functions of the teams like Development, Deployment, Build process, Documentation, Release management etc. yet their specialty lies in Testing. The teams are aware of that and they usually respect that by leaving the testing job to the Testing Specialists.

Some people gets confused with the cross-functional aspect of Agile teams and say that all members test so how come a Tester is special. But good teams know that having a Specialist tester help. In one of the teams that I joined last year, the project lead said to me:

Majd, I don’t want you to spend most of the time writing tests. I’d like to spend most of your time to see how we are doing on the testing front.

So if you are a Tester, you are already a Specialist on testing and you should rather be a T shaped professional by learning the other functions of the team.

The question then comes to developing the other specialty or second leg of a Pi shaped testers. In the webinar mentioned above, when the presenters did few rounds of workshops with testers they found out that the second leg can be Programming, Management, Automation etc. All of those are good but in my opinion for a Tester, knowledge of other fields like Programming or Management actually makes them a better T shaped Tester by expanding breadth of knowledge. A Pi shaped tester’s other leg should also be in testing.

I don’t want to force my opinion and you are free to pick second legs other than a Testing area. But if you take my advice, I’d suggest the other legs could be Performance testing, Security Testing, Usability Testing, Automated Testing etc. By that I mean that you should be subject matter specialist on the subject and not just a Generalist on all. If you want to get started, here is a piece from tester friend Rizwan Jafri that came out this week about automation.

Are you a T shaped or Pi shaped tester? If not, at what stage you are. And if yes, what are the second leg options you’d like to share with us.

“Awesome Testing Discussions” – A meetup in Lahore

Lahore being the cultural capital of Pakistan is best known for it’s festivity. There is a saying in Punjabi that goes like this “sutt din tay utth mailay, kum karaan main kairay waylay” which roughly translates to “There are 7 days in a week and there are 8 Festivals. How can I find any time to do my job?” But on last weekend, we intermixed the technical testing profession topics with the easy going table team discussions so that people can have fun while learning.

The idea was earlier testified few months ago in a meetup that happened in Islamabad and we thought to use the conventional wisdom to repeat things with some variation to newer places in another time. This time around we got great partnership of a technology giant from Lahore Confiz Ltd. who helped us connect so many Knowledge Testers through the independent platform of Pakistan Software Testing Board.

The event took place at a private hall which was specially arranged by the active Confiz testing team lead by Rizwan Jafri. The location provided the conducive environment for the discussion where ten round tables were setup each hosting 5-6 testers along with a Discussion Moderator who helped initiate, expand and summarize the discussion topic for that particular table. The members on each team were selected from our previous Lahore tester meetup participants such that we had testers with experience ranging from <1 years to 14+ years representing over 25 organizations to provide enough diverse opinions.

The planning for the event actually started many weeks before the actual meetup and I’m very much thankful to our tester friends in Lahore: Rizwan Jafri, Kamran Khan, Humaira Anwar, Muhammad Ayyaz and Qaiser Munir who meticulously designed the meetup. The evergreen guidance of Dr. Zohaib Iqbal and Dr. Uzair Khan was always there and Taimur Sarwar being the champion advocator of testing discussion added value by not only in the planning phase but also traveled from Islamabad to join the session with his team members.

The discussion topics were a bit similar to the one we had in Islamabad edition and bit new as introduced by our Discussion Moderators who were all from the Lahore testing community. I’m thankful to Ayyaz and Qaiser who took the additional role of moderation and they were joined by a very talented league of testers that included: Maissom Abbasi, Najabat Ali Khan, Muhammad Ahsan, Sadaf Khan, Ali Khalid, Waqar Ahmad, Maria Mukhtar and Zeeshan Dilawar.

AwesomeTesingDiscussionLahore

(more photos coming soon on our facebook page)

Given the time constraint, the teams were given 35 minutes of discussion time after an opening note by me in which I talked about our philosophy behind holding such meetups. Emphasizing the necessity to push testers to meet new testers, I explained the notion of Echo Chambers and shared some ideas on how to come out of them. I also talked about how just listening to talks won’t mature your thought process unless you express yourself in a safe environment like the one provided by this meetup. To my surprise, my above murmurings about meetups was introduced as a Keynote.

For all of my readers who were not present at the event, below were the ten topics which were then presented by each team:

  1. Becoming a full stack Quality Engineer
  2. Tester-Programmer relationship
  3. Role of tester in User Acceptance Testing
  4. How important is Communication and Collaboration for Testers
  5. I’m bad at coding, can I still be a Tester?
  6. Performance Testing – what is involved when it is well-done
  7. What to and what not to automate
  8. The role of Tester in Agile teams
  9. Considerations in developing an Automation Framework
  10. Should QA Engineer be a Specialist or Generalist?

Each team summed up their discussion through a short presentation of <5 minutes on the stage. The beauty of the plan was that this job was done by the junior most tester on that table which helped our newer lot gain knowledge and confidence side by side. The audience then was invited to throw in questions on the topic or add the comments. The real takeaways were here and I’m sure everyone had a full list of notes from this learning experience to give them new perspectives. To give you an idea of the proceedings, let me share couple of examples.

On the topic of whether a Tester be good at coding, the discussion quickly turned into the balance of Domain knowledge and Technical knowledge that a Tester needs. The comments went in favor of Testers who don’t understand coding but strong at the domain knowledge are better vs. comments going in favor of Testers who know the layers of technical implementation being good. Eventually the hall kind of agreed that we need a mix of both.

Similarly during the discussion on what and what not to automate, some people believed that we need a cautious approach and can only automate within some constraints. But folks with practical knowledge explained that the decision should not be based on how much you know about automation or how much you can achieve, rather it should be based on how much automation is needed by the project you are executing.

These discussions were well summed up by closing note by Omair Sajid, Senior Director Engineering at Confiz who very much liked the idea of testers getting together and explore their profession. He talked through his experience that finding a real Tester has been a challenge and was happy to see that community is raising the standard through such events which will produce quality testers. He also shared his vision of automation and suggested that today’s modern solution developed at high pace for demanding situations cannot succeed until test automation is not applied.

As the certificates for Discussion Moderators and Organizers were being awarded, Dr. Zohaib took the opportunity and shared our future plans to keep this momentum going. It included a definitive plan of a meetup in Karachi in October along with another being planned in Lahore in a few month in collaboration with Systems Ltd. He also outlined the rough plans that we have to hold a national level software testing conference to happen towards end of 2016 which will be a yearly event and will move from one city to another.

Oh I forgot to mention that Lahore along with it’s festivity is known for the food and it was well proved by delicious and mouthwatering eateries that were offered with tea at the end. As the participants enjoyed the tasteful food, the environment resulted in lot of new connections being made on the fly and lots of ideas on how to work together to further build the amazing testing community that we have.

What ideas do you have for our future meetups?

Learning from Errors

My Mama used to say when I was a kid that “Don’t worry. Your mistake is not a mistake, if you learn from it”

In his book “Errors: bugs, boo-boos and blunders”, Jerry Weinberg talks the subject of errors at length and it was a delightful reading. As a software practitioner, we all see errors all the time. But if one wants to take the errors seriously, this book is a must read. To boost your appetite, let me share a few lessons that I learned from the book:

Errors are not a Moral thing

Yes, when we get to know that we were responsible for an error, we feel guilty. Some of it comes from our school time when we were told that making error is a moral thing and the one who does is a bad person. Unfortunately many Managers live in the school teacher mentality where they want to embarrass team members publicly when they make mistakes. Jerry emphasizes this and suggests to get out of this mind block to accept that all of us make errors. This should not result in doing nothing by saying “hey, what we can do as errors are a natural thing” rather this should result in focusing on what we can do to limit the error and how they can be stopped in future. In Jerry’s words:

To err is human; to combat error is software engineering.

Errors have a link with the management

When we study errors and causes of errors, we often focus on the Person who made the error. If we go beyond, we look at what was fed to that Person and how it was done. In terminology, it can be translated as how the requirements were defined, how the Programmer understood the requirements and how the Programmer implemented the requirements. All too often, we forget that the environment in which all these things happen and how it contributes to the errors made. And good management is all about create environment to reduce such errors.

Jerry has listed eight Fs of software failure and how management can work on them. But he also said right there:

But good management is so boring!

Errors will always exist

You might be thinking that you are a software engineering champion and you are a management champion, so you know the best of software processes that you’ll apply in best of your management style such that errors will never occur. That’s the pitfall for each Perfectionist and Jerry suggests that we need to live with the errors. He lists 8 Laws of Errors Defense to help us out and one of them is:

One thing worse than error is an error farm

As a tester, you might have seen many error farms where errors are produced in abundance. When we report errors, we should be looking for error farms and reporting them too.

How do you see errors? What lessons you have learned from them?