Perceiving Quality

(This is a guest post by Arslan * )

The question is well projected and asked from time to time; The question is understood and been worked upon from time to time; The question has seek answers and got them from time to time; but, the question still remains; How do we perceive the right quality for a product where we don’t know who will be the customers? It carries a lot of weight doesn’t it? A weight of actions, responsibilities and care for a product which can satisfy customer; so, how should we go about it?
In his article “Testing without a Map” (2005 January – Better Software), Michael Bolton expressed the use of Quality Criteria Heuristics while testing a product where there are rarely no requirements and specifications mentioned and the tester is like in an ocean and have no map or markers to follow through and reach the mainland. The question is valid till now as it is being asked across the forums and live debates whenever there is a gathering for enthusiastic testers. Michael proposes the use of Quality Criteria Heuristic called “HICCUPPS”, which was introduced by none other than James Bach;

a. History
b. Image
c. Comparable Products
d. Claims
e. User’s Expectations
f. The Product Itself
g. Purpose
h. Statutes


(the original photo is here: )
Now consider yourself into that ocean of “Leaky Abstractions” and tell yourself to test a product or a component of a product where you don’t know what customer wants, what was the intended out come and something to scale the test item in terms of its acceptance; Let’s look at each one of them and then add 3 more in the end to get the idea what this Heuristic can do shall we?

“The features or functions current behavior should be consistent with its past behavior, assuming that there is no good reason for it to change. This heuristic is especially useful when testing a new version of an existing program”

A product is an evolution of models and versions; if there is right kind of customer acceptance then the evolution becomes a necessity, otherwise you need to be consistent enough to give something to the market; The “History” heuristic is about that aspect of a product quality. Meaning, that all new versions of the product must be a reflection to the behavior and reputation of the past versions of the product; How tester would do it; well, try research out what comes in the past; how does the function of “Add” button worked on the past versions, is the same or something else has happened?

“The product’s look and behavior should be consistent with an image that the development organization wants to project to its customers or to its internal users.”

If you are making it then you don’t keep it, you sell it! For serving the latter you need to put up certain expectations against that product. In other words, you need to build up an image of the product for its customers; but is that easy? So the pressure comes on the development organization who then have to consider the consistency of that “image” not only with what they are projecting, but it should also comes as satisfaction for the “Internal” and “External” users; By Internal users we mean the users who use the product but are not part of the development; for example, Testers, Business Analysts and Support professionals;

Comparable Products
“We may be able to use other products as a rough, de facto standard against which our own can be compared.”

Michael Bolton also blogged about this aspect of “Comparable” products and distributed the concept into much details; Comparison works, because once your product is developed and is under testing, implementation or in the hands of your customers it is compared! Question is how and what aspects are there to compare; well here are some for your consideration, we can do comparisons of:

 any product or a software product,
 any attribute of a product or a software product,
 or even attributes of any non-software products

We can also consider these aspects while comparing two or more products:

 An Alternative of the existing product, as in Microsoft Word in Comparison to Apache Open Office
 A Commercially Competitive Product
 A Product member of the same Suite – Try comparing Word with Excel
 Two Products (Features) that are part of the two different product in a suite
 An existing product whose sole purpose is comparable to a specific feature in another product – try Comparing Copy Feature of DOS / Windows to Linux Applications
 An existing product which is different but can be share some similar functionalities; (Usually useful in comparing Input / Process / Output functionality of applications
 Events within the products
 A Reference output of the system in comparison with an output generated from another medium – Calculated results from a Calculator is to results shown on a printed report
 A product we don’t like too much!
 An patterns and behaviours in comparison to the range of the products;

“The product should behave the way some document, artifact, or person says it should”

Quality is perceived but then it is expected the same way. We as testers should know both ends, as what is being perceived by the development, and then what is expected by the customers; what makes this wheel go around is the “Claims” put forwarded by the Marketing and Promotional teams; Product claims are projected via “Word of mouth” or by some kind of publicity artifact or documents; As tester we need to compare and test these claims to be true before letting the customers know about them.

User’s Expectations
“A feature or function should behave in a way that is consistent with our understanding of what users want, as well as with their reasonable expectations.”

This one is not about the explicit requirements, it’s more about the implicit nature of the needs and wants that we deal as product builders; it is never stated anywhere in the requirements that the product will launch itself in two seconds when the icon is clicked, and it is not stated in any meetings that the buttons will function as per their captions, and on edit, save, delete, view, or print the system will not crash! So for user expectations (reasonable) we need to think carefully, use our imagination and common sense and our intuitions to eliminate any such anomaly out of the way which can contradict with the user’s expectations;

Product Itself
“The behavior of a given function should be consistent with the behavior of comparable functions or functional patterns within the same product unless there is a specific reason for it not to be consistent.”

This addresses the internal and external features of the product, or in other terms “The Product Elements” (Structure, Functions, Data, Interfaces, Platform, Operations and Time) – We need the product to fall into the right criteria is because we need it to behave consistently with its features and how they have been constructed and behaving since their birth (by developers). To test that, we need to have past and the present versions available. We need to have proper checklist for potentially important features so we can mark them out as Okay, Improved or Inconsistent to their self being.

“The behavior of a feature, function, or product should be consistent with its apparent purpose.”

Think “Elevators” – now think of using the elevator in a Hotel, then in a Hospital and then in Mining Area; well the function remain same, to get people from one level to another, but how it is constructed actually serves the right purpose. In so many words, the product exist in an environment; a project is constructed within the disciplines of an environment; these factors then construct the purpose of the project; “Customers”, “Information”, “Mission”, “Test Items” etc.

“The product should behave in compliance with legal or regulatory requirements.”

Some people don’t consider that seriously, and with the rapid cultural and technological changes in the current market, the thing “Legal” is losing its very impact. People plagiaries contents like that is their right; but when it comes to large products and big clientele one needs to have good backing of agreements, the NDAs and above all the right behavior, linkage and process mapping of the application in comparison to the legal requirements; Consider the Financial applications, or the compliance systems, if these systems have glitches in their workflows or SOPs, or calculations; the things can go bad to worse within the couple of seconds;

The reason we need to test the application under Quality Criteria is to make sure that we handover a rightly build product to our customers. It is not a one man or one team responsibility; it carries a certain weight of responsibility and experience which is applied and shared upon from one domain to another, until it reaches to where the buck stops; our customers – the satisfied customers.

Do you use HICCUPS or other testing heuristics in testing? Which one is your favorite?

* Arslan Ali is an experienced tester and has been a source of inspiration for the testing community in Pakistan. He very actively manages the largest group of people in this field in Pakistan at his LinkedIn group Software Testers & QA from Pakistan which has over 1400 members (and it is growing). He also conducts trainings on “how to be effective as testers” for both Industry professionals and University Students. His latest activity has been the launch of Outtabox along with Faiza Yousuf. Excellent work guys and keep up the good initiative.


Tags: ,

Trackbacks / Pingbacks

  1. Exploratory Testing vs. ad hoc testing | Knowledge Tester - May 13, 2014

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.