Thoughts on Performance Testing

Every now and then, I’m been involved in discussion with fellow testers on the topic of Performance Testing. I realized that I never covered this on the blog. So here you go.

Note: There are many types of Performance Testing and below notes are related to checking time based performance of systems. Though these apply to other types also, but it’s always good to have a context.

Performance Tests don’t pass or fail

In a way that the result of a test is always a binary: either they pass or fail. Whereas Performance tests give you a number and only you have to decide if that number is reasonable enough to call that a Passing test. For example, you observe that the load time of your website’s home page on IE 10 is 4 seconds. Now is it good or bad performance is always a subjective discussion and unless you set a benchmark, the number looks very bad, bad, ok, good, very good etc.

There can be multiple benchmarks

So as mentioned above, the real question is that against what number we are comparing our test results. Those are called benchmarks.

The benchmarks can be a variety of things including:

  • Your competitor’s similar solution
  • Your last commercial release that is with the client
  • Your weekly/daily alpha release
  • A number that comes out from the person who is responsible for release.
  • A number that comes out of some Usability lab conducted with users.

It is also possible to record actual number and then represent it with compared with one or more or all benchmarks.

performancetesting

(the original photo is here: http://idmdude.com/2014/03/07/its-ok-to-get-stressed-out-with-openam/)

Play around only one variable

Be mindful that performance measurements need to be done in ‘controlled environment’. If you are comparing a number coming from Machine A with Machine B, that is not a comparison as more than variables at play. So rest of the stuff will stay the same serving as baseline and two readings will be taken by changing one variable.

Again looking at the above example of load time of website, you’ll have to check it on the same machine with same memory resources, same set of other processes loaded and then compare the time. So if you are doing a comparison with benchmark, on same machine repeat the test with one or more benchmark. Now you have results that you can compare.

Take multiple measurements

Having one reading can be misleading and may not present the true picture. Depending on the task being measured, take multiple measurements like 5 or 10 or more and then take an average to get the real value.

Tools will help you

There are many tools that will help you measure time or memory. They also help you orchestrate your tests one after another and then provide you with set of numbers. Once you know what kind of testing you want to do, then picking a tool can be easy. For example to check for load time of website, this tool will be handy for Firefox.

Performance tests are in the 4th Quadrant of Agile Testing and are always accelerated by tools. But they are designed by knowledge testers like you.

Did I miss any thing on the general overview of performance testing? What things you’d add to this list?

Tags: ,

7 responses to “Thoughts on Performance Testing”

  1. Ridha Malik says :

    Nice Post (y)

    Like

  2. VADDI SANDEEP says :

    Good one.Can any of you tell me the tools to perform the performance test.
    Advantages too.Thanks in advance.

    Like

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s