Thoughts on Performance Testing

Every now and then, I’m been involved in discussion with fellow testers on the topic of Performance Testing. I realized that I never covered this on the blog. So here you go.

Note: There are many types of Performance Testing and below notes are related to checking time based performance of systems. Though these apply to other types also, but it’s always good to have a context.

Performance Tests don’t pass or fail

In a way that the result of a test is always a binary: either they pass or fail. Whereas Performance tests give you a number and only you have to decide if that number is reasonable enough to call that a Passing test. For example, you observe that the load time of your website’s home page on IE 10 is 4 seconds. Now is it good or bad performance is always a subjective discussion and unless you set a benchmark, the number looks very bad, bad, ok, good, very good etc.

There can be multiple benchmarks

So as mentioned above, the real question is that against what number we are comparing our test results. Those are called benchmarks.

The benchmarks can be a variety of things including:

  • Your competitor’s similar solution
  • Your last commercial release that is with the client
  • Your weekly/daily alpha release
  • A number that comes out from the person who is responsible for release.
  • A number that comes out of some Usability lab conducted with users.

It is also possible to record actual number and then represent it with compared with one or more or all benchmarks.


(the original photo is here:

Play around only one variable

Be mindful that performance measurements need to be done in ‘controlled environment’. If you are comparing a number coming from Machine A with Machine B, that is not a comparison as more than variables at play. So rest of the stuff will stay the same serving as baseline and two readings will be taken by changing one variable.

Again looking at the above example of load time of website, you’ll have to check it on the same machine with same memory resources, same set of other processes loaded and then compare the time. So if you are doing a comparison with benchmark, on same machine repeat the test with one or more benchmark. Now you have results that you can compare.

Take multiple measurements

Having one reading can be misleading and may not present the true picture. Depending on the task being measured, take multiple measurements like 5 or 10 or more and then take an average to get the real value.

Tools will help you

There are many tools that will help you measure time or memory. They also help you orchestrate your tests one after another and then provide you with set of numbers. Once you know what kind of testing you want to do, then picking a tool can be easy. For example to check for load time of website, this tool will be handy for Firefox.

Performance tests are in the 4th Quadrant of Agile Testing and are always accelerated by tools. But they are designed by knowledge testers like you.

Did I miss any thing on the general overview of performance testing? What things you’d add to this list?

Testers, stop saying these 3 things

No one can look inside you to find out how good you are at something. Everyone judges you from what you say and what you do and that is why if you want to build your repute as a Knowledge Tester, you should stop saying these 3 things.

  1. Where are the Specs

When I started learning the craft of software development in late 90’s I liked the quotes that said that “You can only walk on water if it is frozen. And you can only work on requirements if they are frozen”. (I couldn’t find the origin of this; can any one help?)

Fast forward many years and the constant change has made it impossible to freeze the requirements. Our users, their requirement, their priorities, their environments, the technology changes at such a fast pace that freezing them is against the laws of Physics.

So when you start on a new project or feature as a tester, never ask ‘where are the Specs’. Because chances are that either they don’t exist or they are outdated/ out of sync from how the software currently behaves.

I know what you are thinking now that how we can move forward if we don’t know what the software is supposed to do. The answer to that is “find out”. Yes, we can find them out through exploration and invoking Three Amigos conversations with Programmers and Customers.

And if someday you accidentally find the Specs, add them to your list of resources that reveal the ‘truth of expected behavior’. Don’t take them as the absolute resource for this purpose.


(the photo is taken from this article: )

  1. How should I test this?

We often get into this trap that as we engage other members of the team in testing, we ask their advice on how the systems should be tested. There are two problems associated with that kind of statement.

First, people can start doubting your skills by thinking that ‘we hired these guys to test and they ask us how to test. If we knew that, we could have tested this at our own’. The second problem is that you start following the thought process that produced the solution rather than thinking of new ways to look at the problem. Also at times, Programmers give their undone projects to testing team as soon as they are asked how to add more tests.

The right question to ask is: ‘What should I test?’ and then keep on extracting information, absorb that information and then come up with a plan so as how you’ll test it. There is no harm in saying that ‘this is how I plan to test, what thoughts do you have’ as in that case you’ll get answers driven by your test strategy and not following what others suggested you.

  1. I need more time to test

This is a killer statement and if you ask Project Managers about what they hate to hear from testers, this statement will be on the top of the list.

The thing is that testing never ends and it is impossible to find all the time to run all possible scenarios to find all the problems. In fact as you spend more time in testing, you figure out that you need more time for it. See how similar is this to spending time on vacation: you always need more time.

But the fact is that we don’t have time.

So, rather than looking at the equation from testing perspective, we need to look from the time perspective. Set clear rules that if you are given one month to test, this is what you’ll do. If you are given one week to test, this is what you’ll do. And if you are given just one hour, this is what you’ll test. On one of the release decisions that I’m part of, we are down to only 24 hours throughput time on testing. You tell us today you want to release it, within 24 hours you know ‘yes’ or ‘no’.

You might be thinking that I’m promoting automation but in fact I’m not. It is true that quick feedback from testing can be fast tracked through automation but defining clear priorities on what will be tested is what will get you there.

Do you also think that we should stop saying these things as Testers? If no, please let me know why. If yes, what else we should stop saying

State of Unit Testing in Pakistan – Survey results

The results from the survey are in and they are very encouraging.

It suggests that around 60% of the teams have Unit Testing practice in one way or other with around 20% say that they love Unit Testing. By the way, I really liked the Pie chart become in a nice shape and appears to be outcome of a testing report!


As always, the results are not complete until we analyze from where the data came. So here are the notes:

  • There were 40 respondents, so this is in no way true reflection of all Industry. I wish I could get more coverage as I did for the Tester Income. But you know people are more interested in that subject than Unit Testing.
  • Another quick insight into results is that overall survey hits are about double. So half the audience decided not to answer the survey. Either they didn’t want to look bad by saying “No, one does that” or they trusted the other replies.
  • And the result came from all places. 47% were Testers, 32% were Programmers and rest were all scattered. They featured CEOs, CTOs and Business people.
  • There were just a couple replies from outside Pakistan which were not included but I really appreciate the response.

So my friends, as I mentioned that our IT industry is maturing and let’s put our part in making Pakistan a place that produces Quality Software

What is your take away from this survey? And what other topics you would like our blog to survey?


State of Unit Testing in Pakistan – Survey

Pakistan IT industry is on the rise. Some are making way as a brand of their own where as many young ones are emerging. Even the theme of OPEN Islamabad’s Annual event for 2014 was ‘Celebrating Entrepreneurs”.

One sign of a maturing IT industry is that it becomes Quality conscious. Not in the terms of having a Quality Policy or having catchy phrases as company’s motto but following certain practices at the heart of it’s business. The heart of software producers is to “write code” and it only becomes better by taking lot of steps towards excellence. Writing unit tests is one of them.

That is the reason that in recent conversations with my fellows in Islamabad, I hear a lot about their journey towards adapting unit testing. But at times your echo chambers can lead you to a world based on illusions and not on reality. The best way to know the reality is to ask around and that is what Knowledge Tester is doing through this survey just like we found the reality on Tester’s income in Pakistan.

Would you mind filling in this 2 minute survey on the state of Unit Testing in Pakistan:

You can even fill it even if you are from elsewhere by mentioning your country name. And stay tuned for the results that follow in two week’s time.

And can you help me spread this word? Thanks in advance.

Unit testing, what?

On my mission to promote quality, I’m visiting one place after another and talking to one group of people to another. As I talk about Software Quality as a shared responsibility, the concept of “Three Amigos”, Agile testing quadrants, and more; one thing is common: “No one wants to get out of the comfort zone”.

For example, when I suggest Testers to learn the technology they are testing and write some code to automate repetitive testing and testing process, they shy away from writing code. Similarly when I ask Programmers to take some responsibility and start writing Unit testing and try approaches like TDD, the expressions on their faces are not very encouraging. There are many discussions out there and I’m adding some concerns that I recently heard and my attempts to answer them:

“I’m on a project that is in production for multiple years. The code base has matured enough through rounds of testing that adding unit tests won’t add value”

The decision to write unit tests is not based on how many years the code base has been in production but on the basis of how many years the code base will stay in production. If you write some tests today for the bugs found so far in the system, the next deliveries will remain safe from those bugs. Remember the time, when you changed code to fix a bug and another bug poked it’s nose from some other place. If you have tests for them, you have that “safety net” that allows to do modifications to the code with more liberty.

“Well, I’ve tried TDD on our newest project but it seems to be a Productivity killer. I’m afraid that TDD doesn’t work well on green field projects”

In fact Test Driven Development is more suited for green field projects.

My experience being a Tester, the receiving end of Quality is that:

The bugs found in projects with TDD/Unit testing is about half compared to Projects with no Unit tests.

The reason you are seeing some productivity loss is that you are seeing the investment in writing tests as a repetitive investment where as it is a diminishing investment. As the time passes by and you run your set of tests again and again, the time needed to maintain existing tests/ add new tests will be lesser. And more importantly the “unknown” time that will be added to the project when testing team finds bugs (not covered in unit tests) is actually being planned into your work automatically.

Let’s take the following as example:

Project A incorporates TDD practice and this is how the time is spent in Coding vs. Testing+Debugging where blue is coding and yellow is the other part. Where as Project B doesn’t incorporate TDD and this is how it’s time is spent over a period of four months. These are hypothetical times but from experience, I can tell that the Yellow time remains mostly smaller in projects like Project A compared to the likes of Project B.

Unit Testing vs No Unit Testing

“So you are saying that as a Programmer, I write tests for my own code. So if I write tests to verify my understanding of the system, how those tests will add value?”

Right and you’ll be surprised of how your understanding starts improving as you write tests. Actually when you write tests, you more and more think like “how the user will use this feature?” rather than “how should I design the feature?” This brings in added checks and improved functionality.

What concerns do you hear and how do you answer them?

Tester Diaries – Applying SFDPOT Heuristic

(This is a guest post by Ridha*)

Welcome back to the Tester diaries. As you know that I am a student of BS in Software Engineering and learning software testing in interesting ways.

Now a days, I am working on my Final Year Project (FYP) and I wanted to design its test cases in parallel with development. Through the Knowledge Tester’s guidance, I came to know about SFDPOT and I was suggested to use this thought provoking heuristic for designing tests. As we all know heuristics not always find the best solution but these are some tricks to make our work easy.

SFDPOT letters stand for Structure, Function, Data, Platform, Operations and Time.  Read in detail about SFDPOT at: and have a look at one example here: . (Note: the latest version of this heuristic is now SFDiPOT where i stands for Interface)

SFDPOT helped me to organize my understanding of testing my FYP in form of a mind map. My FYP name is CureJunction and I have plans to make it live one day at . This will be a website that helps us find doctors with their specialty in our locality and then let us book online appointment with them. Many more functionalities will be provided at this website and I categorized them in the mind map that is shown below. It took me four sheets of paper and a multiple of four hours to come up with the final version :)


Do let me know how it is helpful for you? And how I can make it even better?

(Ridha Malik is becoming a Knowledge Tester while completing her BS in Software Engineering from University of Engineering & Technology, Taxila, Pakistan. As she explores the field, she is sharing her experiences as Tester Diaries and this is second edition of her thoughts. The first one was about Exploratory Testing.)

Recapping 2014

The stats helper monkeys prepared a 2014 annual report for this blog. I’m very thankful to all my visitors and dedicate this report to them!

Here’s an excerpt:

The concert hall at the Sydney Opera House holds 2,700 people. This blog was viewed about 14,000 times in 2014. If it were a concert at Sydney Opera House, it would take about 5 sold-out performances for that many people to see it.

Click here to see the complete report.


Get every new post delivered to your Inbox.

Join 880 other followers